CSC128 : Introduction to UNIX

Using sed


The sed Utility

The sed utility is a stream editor.   Just as vi lets you edit files interactively, sed lets you write a special sed script that controls how the files specified are edited.

Usage:
sed [-n] -f command-file [file-list]
sed [-n] command [file-list]


If [file-list ] is not specified, sed reads from stdin.  

By default, sed writes to stdout. To make sed only copy lines to stdout as specified by the p command,  use the -n option.

If the -f option is used, then commands are read from command-file , a text file containing one or more sed commands. It is important to note that the file, since it is an argument to sed, is not required to be executable, and the first line of a sed script does not need to begin with the typical #!/bin/bash of a shell script.

The command format is:
[address range]instruction[argument-list ]
[address range] specifies what line(s) the instruction should be applied to.
(If no
[address range] is specified, then instruction is applied to all lines.)
Examples:
line 1 only
1

lines 2-10 only
2,10

the last line only
$

line 25 to the last line only
25,$

all lines matching the Regular Expression "regex"
/regex/

all lines except lines 2-10
2,10!


instruction[argument-list] specifes what action should be taken on lines specified in the [address range] .
d
deletes the line from the output (not in the original file)

p
print the line to stdout, even if -n has been specified on the command line.

w filename
write the line to the file named filename

r filename
append the contents of filename after the line(s).

s/regex/replacement/
substitute the first match for the Regular Expression regex with the text replacement on each line.  If an unescaped ampersand (& ) appears in the replacement ,  it will be expanded to the matched string.

s/regex/replacement/g
substitute all matches for the Regular Expression regex with the text replacement on each line. If an unescaped ampersand (& ) appears in the replacement ,  it will be expanded to the matched string.

a\
text
appends the specified text after the selected line(s).  (This can only be used from a sed  command file.)

i\
text
inserts the specified text before the selected line(s) (This can only be used from a sed command file.)

c\
text
changes the line(s) to the specified text.   (This can only be used from a sed command file.)

Examples:

In this following example,  lines 1 through 3 are deleted from the output:
user@machine:~ $ cat man_from_esser
There once was an old man of Esser,
Whose knowledge grew lesser and lesser,
It at last grew so small
He knew nothing at all,
And now he's a college professor.


user@machine:~ $ sed '1,3d' man_from_esser
He knew nothing at all,
And now he's a college professor.


user@machine:~ $



In this following example, all lines containing a match to the Regular Expression ",$" are printed to output.
user@machine:~ $ sed '/,$/p' man_from_esser
There once was an old man of Esser,
There once was an old man of Esser,
Whose knowledge grew lesser and lesser,
Whose knowledge grew lesser and lesser,
It at last grew so small
He knew nothing at all,
He knew nothing at all,
And now he's a college professor.


user@machine:~ $


Note that printed lines appear twice in the above example, since every line is output by default, and then output again by the print command.  To only output lines specifically printed, use the
-n option:
user@machine:~ $ sed -n '/,$/p' man_from_esser
There once was an old man of Esser,
Whose knowledge grew lesser and lesser,
He knew nothing at all,


user@machine:~ $


In this following example, on line 5 only, the first match of the Regular Expression
"e" is replaced with "E" .
user@machine:~ $ sed '5s/e/E/' man_from_esser
There once was an old man of Esser,
Whose knowledge grew lesser and lesser,
It at last grew so small
He knew nothing at all,
And now hE's a college professor.


user@machine:~ $


In this following example, on line 5 only, all matches of the Regular Expression
"e" is replaced with "E" .
user@machine:~ $ sed '5s/e/E/g' man_from_esser
There once was an old man of Esser,
Whose knowledge grew lesser and lesser,
It at last grew so small
He knew nothing at all,
And now hE's a collEgE profEssor.


user@machine:~ $


In this following example, on line 5 only, all matches of the Regular Expression "n.." is preceded with "INSERT" . (The ampersand (& ) is expanded to the matched string.)
user@machine:~ $ sed '5s/n./INSERT&/g' man_from_esser
There once was an old man of Esser,
Whose knowledge grew lesser and lesser,
It at last grew so small
He knew nothing at all,
AINSERTnd INSERTnow he's a college professor.


user@machine:~ $


In this following example,  a
sed command file is used to append text after line 1:
user@machine:~ $ cat commands.sed
1a\
APPENDED TEXT

user@machine:~ $ sed -f commands.sed man_from_esser
There once was an old man of Esser,
APPENDED TEXT
Whose knowledge grew lesser and lesser,
It at last grew so small
He knew nothing at all,
And now he's a college professor.

user@machine:~ $


In this following example,  a sed command file is used to insert text before each line.  Since no address range is specified, the command applies to all lines:
user@machine:~ $ cat commands.sed
i\
INSERTED TEXT


user@machine:~ $ sed -f commands.sed man_from_esser
INSERTED TEXT
There once was an old man of Esser,
INSERTED TEXT
Whose knowledge grew lesser and lesser,
INSERTED TEXT
It at last grew so small
INSERTED TEXT
He knew nothing at all,
INSERTED TEXT
And now he's a college professor.


user@machine:~ $


In this following example,  a sed command file is used to change the text of line 3 and delete the line with a match to the Regular Expression "once" .  Notice that line three is replaced with all the text specified in the command file, with backslashes (\ )  indicating that the following line is also part of the text, another command.  The last line of the text does not end with a backslash.   (Any number of commands can be in the command file.)
user@machine:~ $ cat man_from_esser
There once was an old man of Esser,
Whose knowledge grew lesser and lesser,
It at last grew so small
He knew nothing at all,
And now he's a college professor.


user@machine:~ $ cat commands.sed
3c\
THIS IS A LINE\
THIS IS ANOTHER LINE
/once/d

user@machine:~ $ sed -f commands.sed man_from_esser
Whose knowledge grew lesser and lesser,
THIS IS A LINE
THIS IS ANOTHER LINE
He knew nothing at all,
And now he's a college professor.


user@machine:~ $




Toolbox

diff
Displays the differences between two files.  If the files are different, the exit code will be set to non-zero;  if the files are the same, the exit code will be set to zero.
Usage:

diff [options] file1 file2


Options:

-q
quiet.  When this option is specified, nothing is output unless the files differ, in which case a short message to that effect is generated.

--ed
create an ed script that will edit file1 to be the same as file2 .  To use the ed script, edit it to include a write (w ) and quit (q ) commands, and redirect it into the stdin of ed file1 .


Using diff and ed (the editor) can be used to save only the changes to a file, much like rsync or a Revision Control System greatly reducing the space required for backups.

Here is an Example of using ed and diff to save a file's changes and restore a file to it's original contents

Follow along with this Example, commands to be typed are in monospace font:


mkdir diff
Makes a directory for this example

cd diff
Changes you into the new directory

cp ~smauney/public/poem ./poem1
Copy a file to the new directory

cp poem1 poem2
Make a copy of that file


vi poem1
(make some changes to poem1 here)

diff --ed poem1 poem2 >edscript
this uses the --ed option to diff which makes it create an ed script (note redirection to a script file)

cat >> edscript [enter] w [enter] q [enter] ctrl-d
This command uses cat to directly append to the script file a w and a q each on their own line and a Control-D which is the end of file character which tells ed that the script is done

ls -l
(this will show that the 2 poem files are different, you can also run
diff poem1 poem2 and see the differences)

ed poem1 <edscript
This places the changes back into poem1

diff poem1 poem2 This shows no difference between the 2 files now (ls -l will also show that the files have the same size)

There is also a utility called patch that can be used to re-construct a file.