sed: Stream Editing Mastery

Why This Matters

You need to change a configuration value across 200 files. Or strip all comments from a config before processing it. Or reformat a CSV export that has dates in the wrong format. Or fix a typo in a thousand log entries being piped through a pipeline.

You are not going to open each file in a text editor. You are going to use sed.

sed (stream editor) reads input line by line, applies transformations, and writes the result to standard output. It does not load the entire file into memory, so it can process files of any size. It works in pipelines, so it fits naturally into the Unix tool chain. And it can edit files in-place, making batch modifications trivial.

If grep finds text, sed transforms it.

Try This Right Now

# Simple substitution
echo "Hello World" | sed 's/World/Linux/'
# Hello Linux

# Delete lines containing a pattern
echo -e "keep this\ndelete this line\nkeep this too" | sed '/delete/d'
# keep this
# keep this too

# Print only lines matching a pattern (like grep)
echo -e "ERROR: something\nINFO: normal\nERROR: another" | sed -n '/ERROR/p'
# ERROR: something
# ERROR: another

# In-place edit of a file (with backup)
echo "color=red" > /tmp/test.conf
sed -i.bak 's/red/blue/' /tmp/test.conf
cat /tmp/test.conf      # color=blue
cat /tmp/test.conf.bak  # color=red
rm /tmp/test.conf /tmp/test.conf.bak

How sed Works

sed processes input one line at a time through this cycle:

+------------------------------------------------------------------+
|  1. Read a line from input into the "pattern space"              |
|  2. Apply all commands (in order) to the pattern space           |
|  3. Print the pattern space to stdout (unless -n is used)        |
|  4. Clear the pattern space                                       |
|  5. Repeat for the next line                                      |
+------------------------------------------------------------------+

    Input        Pattern Space       Commands        Output
  +-------+     +------------+     +----------+    +--------+
  | Line 1| --> |  "Line 1"  | --> | s/old/new| -> | result |
  | Line 2|     |            |     | /pat/d   |    |        |
  | Line 3|     |            |     | ...      |    |        |
  +-------+     +------------+     +----------+    +--------+

The basic syntax is:

sed [options] 'commands' [input-file...]

# Or in a pipeline:
command | sed 'commands'

Substitution: s///

The s command is what you will use 90% of the time. It replaces text matching a pattern with a replacement.

Basic Substitution

# Replace first occurrence on each line
echo "cat cat cat" | sed 's/cat/dog/'
# dog cat cat (only the first "cat" changed)

# Replace ALL occurrences on each line (global flag)
echo "cat cat cat" | sed 's/cat/dog/g'
# dog dog dog

# Replace the Nth occurrence
echo "cat cat cat cat" | sed 's/cat/dog/3'
# cat cat dog cat (only the 3rd)

Substitution Flags

Flag	Meaning
`g`	Replace all occurrences on the line (global)
`p`	Print the line if a substitution was made
`i` or `I`	Case-insensitive match
`n`	Replace only the Nth occurrence
`w file`	Write matched lines to a file

# Case-insensitive substitution
echo "Hello HELLO hello" | sed 's/hello/hi/gi'
# hi hi hi

# Print only lines where substitution happened
echo -e "good line\nbad line\ngood line" | sed -n 's/bad/fixed/p'
# fixed line

Using Different Delimiters

When your pattern contains /, use a different delimiter to avoid escaping:

# Awkward with / delimiter
sed 's/\/home\/user/\/opt\/app/' file

# Much cleaner with | or # as delimiter
sed 's|/home/user|/opt/app|' file
sed 's#/home/user#/opt/app#' file

You can use almost any character as the delimiter.

The Replacement String

Special sequences in the replacement:

Sequence	Meaning
`&`	The entire matched text
`\1`-`\9`	Backreference to capture group
`\U`	Uppercase everything that follows (GNU sed)
`\L`	Lowercase everything that follows (GNU sed)
`\u`	Uppercase only the next character (GNU sed)
`\l`	Lowercase only the next character (GNU sed)

# & refers to the whole match
echo "hello world" | sed 's/[a-z]*/(&)/g'
# (hello) (world)

# Backreferences with capture groups
echo "John Smith" | sed 's/\([A-Z][a-z]*\) \([A-Z][a-z]*\)/\2, \1/'
# Smith, John

# Same with ERE (-E flag, cleaner syntax)
echo "John Smith" | sed -E 's/([A-Z][a-z]+) ([A-Z][a-z]+)/\2, \1/'
# Smith, John

# Case conversion (GNU sed)
echo "hello world" | sed 's/.*/\U&/'
# HELLO WORLD

echo "HELLO WORLD" | sed 's/.*/\L&/'
# hello world

# Capitalize first letter of each word
echo "hello world" | sed -E 's/\b([a-z])/\u\1/g'
# Hello World

Think About It: When would you use & versus \1 in a replacement? Think about when you want the entire match versus just part of it.

Addresses: Targeting Specific Lines

By default, sed commands apply to every line. Addresses restrict which lines a command operates on.

Line Number Addresses

# Only line 3
sed '3s/old/new/' file

# Lines 2 through 5
sed '2,5s/old/new/' file

# First line
sed '1s/old/new/' file

# Last line
sed '$s/old/new/' file

# Every other line (starting from line 1, step 2)
sed '1~2s/old/new/' file

Pattern Addresses

# Lines matching a pattern
sed '/ERROR/s/old/new/' file

# Lines NOT matching a pattern
sed '/ERROR/!s/old/new/' file

# Between two patterns (inclusive)
sed '/START/,/END/s/old/new/' file

Combining Address Types

# From line 5 to the first line matching "END"
sed '5,/END/s/old/new/' file

# From line matching "START" to line 10
sed '/START/,10s/old/new/' file

Delete, Insert, and Append

Delete Lines: d

# Delete lines matching a pattern
sed '/^#/d' /etc/ssh/sshd_config
# Remove all comment lines

# Delete blank lines
sed '/^$/d' file

# Delete comment lines AND blank lines
sed '/^#/d; /^$/d' file

# Delete lines 1 through 5
sed '1,5d' file

# Delete the last line
sed '$d' file

# Delete everything EXCEPT lines matching a pattern
sed '/important/!d' file
# (equivalent to grep "important")

Insert Lines: i

Insert text before a line:

# Insert before line 3
sed '3i\This is inserted before line 3' file

# Insert before lines matching a pattern
sed '/ERROR/i\--- Error found below ---' file

Append Lines: a

Append text after a line:

# Append after line 3
sed '3a\This is appended after line 3' file

# Append after the last line
sed '$a\This is the final line' file

# Add a blank line after each line (double-space)
sed 'a\\' file

Change Lines: c

Replace entire lines:

# Replace line 3 entirely
sed '3c\This replaces line 3 completely' file

# Replace lines matching a pattern
sed '/old_setting/c\new_setting=true' config.ini

In-Place Editing: -i

The -i flag modifies files directly instead of writing to stdout.

# Edit in-place (no backup -- dangerous!)
sed -i 's/old/new/g' file.txt

# Edit in-place WITH backup
sed -i.bak 's/old/new/g' file.txt
# Creates file.txt.bak with original content

WARNING: sed -i modifies files permanently. Always test your command without -i first, or use -i.bak to keep a backup. There is no undo.

Distro Note: On macOS/BSD, sed -i requires an argument (even empty): sed -i '' 's/old/new/' file. On GNU/Linux, sed -i 's/old/new/' file works without an argument. For cross-platform scripts, use sed -i.bak which works on both.

In-Place Editing Multiple Files

# Change "foo" to "bar" in all .conf files
sed -i.bak 's/foo/bar/g' /etc/myapp/*.conf

# Remove backup files after verifying
diff /etc/myapp/main.conf /etc/myapp/main.conf.bak
rm /etc/myapp/*.bak

Multiple Commands

Using -e

sed -e 's/cat/dog/' -e 's/red/blue/' file

Using Semicolons

sed 's/cat/dog/; s/red/blue/' file

Using a Script File

For complex transformations, put commands in a file:

cat > /tmp/transform.sed << 'SED'
# Remove comments
/^#/d

# Remove blank lines
/^$/d

# Trim trailing whitespace
s/[[:space:]]*$//

# Replace tabs with spaces
s/\t/    /g
SED

sed -f /tmp/transform.sed input.txt

Hands-On: Practical sed Examples

Setup

cat > /tmp/sed-lab.txt << 'DATA'
# Server Configuration
# Last updated: 2025-03-01

server_name = production-web-01
listen_port = 8080
max_connections = 100
log_level = debug

# Database settings
db_host = 10.0.1.50
db_port = 5432
db_name = myapp_prod
db_user = admin
db_password = secret123

# Feature flags
enable_cache = true
enable_debug = true
DATA

Example 1: Clean Config (Remove Comments and Blank Lines)

sed '/^#/d; /^$/d' /tmp/sed-lab.txt

Output:

server_name = production-web-01
listen_port = 8080
max_connections = 100
log_level = debug
db_host = 10.0.1.50
db_port = 5432
db_name = myapp_prod
db_user = admin
db_password = secret123
enable_cache = true
enable_debug = true

Example 2: Change a Configuration Value

# Change log_level from debug to info
sed 's/log_level = debug/log_level = info/' /tmp/sed-lab.txt

More robust (handles varying whitespace):

sed -E 's/(log_level\s*=\s*).*/\1info/' /tmp/sed-lab.txt

Example 3: Comment Out a Line

# Comment out the debug setting
sed '/enable_debug/s/^/# /' /tmp/sed-lab.txt

Example 4: Add a Setting After a Section Header

# Add a timeout setting after the database section header
sed '/# Database settings/a\db_timeout = 30' /tmp/sed-lab.txt

Example 5: Extract Values

# Extract just the database host value
sed -n '/^db_host/s/.*= //p' /tmp/sed-lab.txt
# 10.0.1.50

Example 6: Multiple Transformations for Deployment

# Prepare config for staging environment
sed -E \
    -e 's/(server_name\s*=\s*).*/\1staging-web-01/' \
    -e 's/(db_name\s*=\s*).*/\1myapp_staging/' \
    -e 's/(enable_debug\s*=\s*).*/\1false/' \
    -e 's/(log_level\s*=\s*).*/\1warning/' \
    /tmp/sed-lab.txt

The Hold Space

sed has two buffers: the pattern space (where the current line lives) and the hold space (a secondary buffer for storing text between lines).

Command	Action
`h`	Copy pattern space to hold space (overwrite)
`H`	Append pattern space to hold space
`g`	Copy hold space to pattern space (overwrite)
`G`	Append hold space to pattern space
`x`	Exchange pattern space and hold space

The hold space is advanced, but here is a practical example:

# Reverse the order of lines in a file
sed -n '1!G;h;$p' file

# This is equivalent to the tac command:
tac file

How it works:

1!G -- For every line except the first, append hold space to pattern space
h -- Copy pattern space to hold space
$p -- On the last line, print the pattern space

Another practical use -- print a line and the line before it:

# Print the line before each ERROR line (gives context)
sed -n '/ERROR/{x;p;x;p;};h' /tmp/practice.log 2>/dev/null || true

For most practical tasks, you will not need the hold space. The pattern space and regular commands handle 95% of use cases.

Debug This: sed Substitution Not Working

You try to uncomment a line in a config file:

sed 's/^#listen_port/listen_port/' /tmp/sed-lab.txt

But nothing changes. The line still has the #.

Diagnosis:

# Look at the actual line
grep "listen_port" /tmp/sed-lab.txt
# listen_port = 8080

The line is not commented out. There is no # before listen_port. Your pattern does not match anything, so nothing changes. sed does not produce an error when a pattern does not match -- it just leaves the line unchanged.

Another common issue:

# Trying to replace a path
sed 's/home/user/opt/app/' file
# ERROR: unknown option to 's' command

The / in the path conflicts with the / delimiter. Fix it by using a different delimiter:

sed 's|home/user|opt/app|' file

sed debugging tips:

Always test without -i first -- let sed print to stdout
Use -n with p to see which lines match: sed -n '/pattern/p'
When in doubt, print the file and look at the actual content
Use a different delimiter when patterns contain /
Remember: BRE by default, add -E for extended regex

What Just Happened?

+------------------------------------------------------------------+
|                     CHAPTER 21 RECAP                              |
+------------------------------------------------------------------+
|                                                                  |
|  - sed processes input line-by-line through the pattern space    |
|  - s/old/new/g is the workhorse command (substitute)            |
|  - Use g flag for all occurrences, i flag for case-insensitive  |
|  - Addresses target specific lines: numbers, patterns, ranges   |
|  - d deletes lines, i inserts before, a appends after           |
|  - -i edits files in-place (use -i.bak for safety)              |
|  - Use | or # as delimiter when patterns contain /              |
|  - -E enables extended regex (cleaner syntax)                    |
|  - & in replacement = entire match; \1 = first capture group    |
|  - Always test without -i before modifying files                 |
|                                                                  |
+------------------------------------------------------------------+

Try This

Exercise 1: Config File Editing

Take a copy of /etc/ssh/sshd_config and use sed to:

Remove all comment lines and blank lines
Change #Port 22 to Port 2222 (uncomment and change)
Change #PermitRootLogin to PermitRootLogin no

Exercise 2: Log Processing

Using the practice log from Chapter 20, use sed to:

Replace all IP addresses with [REDACTED]
Convert all timestamps from HH:MM:SS to just HH:MM
Remove all INFO lines, keeping only WARN, ERROR, and FATAL

Exercise 3: Batch File Rename

Create 10 files named report_2024_01.txt through report_2024_10.txt. Use a combination of ls and sed to generate mv commands that rename them to report_2025_01.txt through report_2025_10.txt. Pipe the output to bash to execute the renames.

Exercise 4: Data Reformatting

Create a CSV file with names in "First Last" format. Use sed to convert them to "Last, First" format.

Bonus Challenge

Write a sed script (using -f) that takes an HTML file and strips all HTML tags, converting it to plain text. Test it on a simple HTML file you create.

Linux Book: From First Boot to Production