Pipes, Redirection and Text Processing
Connect commands with pipes, redirect output and errors, and analyse a real web server log with cut, sort, uniq, awk, sed, tr and tee.
On this page 11 sections
Introduction
The Unix philosophy is to write small programs that each do one thing well, then combine them. The pipe character | makes that possible: it sends the output of one command straight into the next. In this lesson you will use pipes to answer real questions about a web server’s access log — which visitors are most active and who keeps requesting missing pages.
- How pipes connect commands.
- How to extract and count data with cut, sort and uniq.
- How to filter columns with awk and change text with sed and tr.
- How to redirect errors, discard output and save results with tee.
Each line of the log records one request: the visitor’s IP address, the time, the request, the status code (200 is success, 404 is not found, 500 is a server error) and the size:
head -n 3 access.logClick Run to see what this code prints.
The Pipe |
wc -l counts lines in a file. With a pipe, it counts lines coming from another command instead. How many requests hit a missing page?
grep "404" access.log | wc -lClick Run to see what this code prints.
grep found the 404 lines and passed them to wc, which counted them. You can chain as many commands as you like — each one transforms the stream a little more.
Cutting Out Columns
cut extracts columns. -d ' ' says columns are separated by spaces and -f 1 takes the first one, the IP address:
cut -d ' ' -f 1 access.log | head -n 4Click Run to see what this code prints.
sort and uniq
uniq -c collapses identical neighbouring lines and counts them, so you always sort first to put identical lines together:
cut -d ' ' -f 1 access.log | sort | uniq -cClick Run to see what this code prints.
Sort that result numerically (-n) in reverse (-r) and you have a ranking of the most active visitors — a pipeline of four small commands:
cut -d ' ' -f 1 access.log | sort | uniq -c | sort -rnClick Run to see what this code prints.
Build the pipeline that ranks the most requested pages. Put the stages in order.
- awk '{print $7}' access.log
- sort -rn
- sort
- head -n 5
- uniq -c
awk: Columns and Conditions
awk splits each line into fields named $1, $2, $3 and so on, and can filter with conditions. The status code is field 9:
awk '{print $9}' access.log | sort | uniq -cClick Run to see what this code prints.
Add a condition to show only the 404 requests, printing the IP ($1) and page ($7):
awk '$9 == 404 {print $1, $7}' access.logClick Run to see what this code prints.
Interesting: one visitor requested /wp-login.php and /.env — pages this site never had. That is typical of automated bots probing for weaknesses, and exactly the kind of thing log analysis reveals.
sed and tr: Changing Text
tr translates characters, for example lower case to upper case:
echo "hello linux" | tr "a-z" "A-Z"Click Run to see what this code prints.
sed (stream editor) finds and replaces text with s/old/new/. Slashes inside the pattern are escaped with backslashes:
sed 's/HTTP\/1.1/HTTP\/2/' access.log | head -n 2Click Run to see what this code prints.
By default sed prints the changed text and leaves the file alone. sed -i edits the file in place — useful in scripts, but make a copy first.
Output, Errors and tee
Every command has two output streams: standard output (stdout, number 1) for results and standard error (stderr, number 2) for error messages. You can redirect them separately:
ls notes.txt missing.txt > found.txt 2> errors.txtClick Run to see what this code prints.
| Syntax | Meaning |
|---|---|
| command > file | Send output to a file (replacing it) |
| command >> file | Append output to a file |
| command 2> file | Send error messages to a file |
| command > file 2>&1 | Send both output and errors to the same file |
| command 2> /dev/null | Throw error messages away |
| command < file | Read input from a file |
/dev/null is a special file that discards anything written to it. Every command also leaves an exit status in $? — 0 means success, anything else means failure:
ls missing.txt 2> /dev/nullecho "exit status: $?"Click Run to see what this code prints.
tee is a T-junction: it writes the stream to a file and passes it on down the pipe at the same time:
grep " 404 " access.log | tee not-found.log | wc -lcat not-found.logClick Run to see what this code prints.
A nightly job should save both its normal output and any error messages to job.log. Which command does that?
Common Mistakes
- Running uniq without sort first, so identical lines that are not adjacent are counted separately.
- Sorting counts without -n, so 10 comes before 9 (text order instead of number order).
- Writing > input.txt in the same pipeline that reads input.txt — the file is emptied before it is read.
- Hiding all errors with 2>/dev/null and then missing a real problem.
Frequently Asked Questions
Not at first. Printing columns ($1, $7) and simple conditions ($9 == 404) cover most everyday use. awk is a whole programming language if you ever need more.
Send stream 2 (errors) to wherever stream 1 (output) currently goes. Its position matters: put it after > file.
For quick questions, very often. For large systems teams also use tools like Elasticsearch, Loki or cloud logging services, but command-line pipelines remain the fastest first look.
Key Takeaways
- The pipe | sends one command’s output into the next command.
- cut extracts columns; sort then uniq -c counts; sort -rn ranks by number.
- awk prints fields ($1, $7) and filters with conditions; sed replaces text; tr translates characters.
- > and >> redirect output, 2> redirects errors, 2>&1 combines them, /dev/null discards.
- tee saves a stream to a file while passing it on; $? holds the last exit status.
Summary
You can now build powerful one-line tools out of small commands — the heart of working on the command line. Next you will look at the programs running on your system: processes, background jobs and how to stop them.