grep, sed, awk: Master Text Processing on Linux

grep, sed, awk are the holy trinity of Linux text processing. grep finds patterns, sed modifies text, awk extracts and formats data. Master these three and you control text on Linux. Essential for logs, configs, data parsing.

Tools Overview

grep: Find Patterns

grep "error" /var/log/syslog
grep -i "ERROR" /var/log/syslog          # Case insensitive
grep -c "error" /var/log/syslog          # Count matches
grep -n "error" /var/log/syslog          # Show line numbers
grep -E "error|warning" /var/log/syslog  # Multiple patterns (extended regex)
grep -v "error" /var/log/syslog          # Invert (lines WITHOUT match)

sed: Find and Replace

sed 's/old/new/' file.txt          # Replace first on each line
sed 's/old/new/g' file.txt        # Replace all occurrences
sed -i 's/old/new/g' file.txt     # In-place edit
sed '5d' file.txt                  # Delete line 5
sed -n '5,10p' file.txt            # Print lines 5-10 only

awk: Data Extraction

awk '{print $1}' file.txt         # Print first column
awk '{print $1, $3}' file.txt     # Print columns 1 and 3
awk -F: '{print $1}' /etc/passwd  # Use colon separator
awk '{sum += $1} END {print sum}' file.txt  # Sum column

grep Examples

# Find failed login attempts
grep "Failed password" /var/log/auth.log

# Count PHP errors by type
grep -o 'PHP Parse error.*' /var/log/php.log | sort | uniq -c

# Find MySQL slow queries
grep "Query_time" /var/log/mysql/slow.log

sed Examples

# Replace all IPs in config
sed -i 's/192.168.1.1/192.168.2.1/g' /etc/config.conf

# Remove blank lines
sed '/^$/d' file.txt

# Add line numbers
sed = file.txt | sed 'N;s/\n/ /'

awk Examples

# Parse Apache access log (IP field)
awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c

# Extract database size from MySQL status
mysql -e "SHOW DATABASES;" | awk '{print $1}'

# Sum file sizes in directory
ls -la | awk '{sum += $5} END {print sum / 1024 / 1024 " MB"}'

Combining Tools

# Find errors, count by type, show top 10
grep "ERROR" /var/log/syslog | awk '{print $NF}' | sort | uniq -c | sort -rn | head -10

# Parse log, extract IPs failing auth
grep "Failed password" /var/log/auth.log | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | sort | uniq -c

# Find large files modified today
find / -type f -mtime 0 -exec ls -lh {} \; | awk '$5 > "100M" {print $0}'

Related: Bash scripting | Command-line basics | Text processing | Log analysis

Related articles