Module 1 · Lesson 7 of 28

Pipes, redirection, grep and regular expressions

Redirect the three streams, chain one-purpose tools with pipes, master grep and regular expressions, and rank a web log's top IPs.

  • Beginner to advanced
  • 22 min read
  • 40 min lab
What you will need

Ubuntu 24.04 (WSL2, VM or container); a Raspberry Pi for the embedded sections

By the end you can 7 objectives
  • Redirect stdout, stderr and stdin independently with >, >>, <, 2> and &>, and predict where each stream lands.
  • Explain why 2>&1 >file and >file 2>&1 differ, and describe how a pipe connects stdout to stdin.
  • Filter lines with grep using basic patterns plus -c, -v, -i and -w, and explain that grep matches patterns, not words.
  • Write basic and extended regular expressions using anchors, character classes, quantifiers and alternation.
  • Reshape text with cut, sort, uniq -c and tr, and explain why uniq needs sorted input.
  • Compose a pipe chain that reports the top client IPs and the 404 count from a web-server access log.
  • Describe the kernel's role in a pipe, the buffer, blocking and SIGPIPE, and why stdout is buffered but stderr is not.
Read the lesson below; the video will appear here when published.
01

Three streams and file descriptors

3 min read

Every process on Linux starts life with three open channels. They are called the standard streams, and they are referred to by number, not by name. Standard input is file descriptor 0, standard output is file descriptor 1, and standard error is file descriptor 2.

By default all three point at your terminal. When you type into bash, the shell reads from stdin. When a command prints a result, it writes to stdout. When it reports a problem, it writes to stderr. The two output streams look identical on screen, but they are separate channels, and that separation is the whole point of this lesson.

bash
ls /etc/passwd    # writes the path to stdout
ls /nope          # writes an error to stderr

Both lines appear in your terminal, which hides the difference. Typical output (yours will differ) looks like two normal lines, but the first came from fd 1 and the second from fd 2. You want that split because you usually save results to a file while keeping errors visible in front of you.

You can inspect a running process's descriptors through /proc. The shell's own descriptors show up under /proc/$$/fd:

bash
ls -l /proc/$$/fd

Every tool from cat to a long-running daemon follows the same three-descriptor convention. A headless board, with no one watching a screen, depends on it: a service logs results to one place and faults to another, and the two never mix.

Why two output streams at all? A single stream cannot separate an answer from a warning about how the answer was produced. Unix has kept results and diagnostics apart since the 1970s for exactly this reason, and the numbering 0, 1 and 2 is fixed by convention, not by hardware. The terminal is itself a file, usually /dev/tty, and all three descriptors simply point at it until you redirect them. On a board with no monitor they point instead at a serial console such as /dev/ttyAMA0 or /dev/ttyS0, which is why the serial connection you used to reach your Raspberry Pi shows the same streams.

The three standard streamsstdinstdoutstderrFile descriptor012Default targetterminalkeyboardterminalscreenterminalscreenUsed forinputresultsdiagnostics
The three standard streams and their file descriptors, default targets and purposes.
Three streams, three file descriptors
learner@embeddex:~/lab$ # a successful lookup writes the path to stdout
learner@embeddex:~/lab$ ls /etc/passwd
/etc/passwd
learner@embeddex:~/lab$ # a failure writes the error to stderr, not stdout
learner@embeddex:~/lab$ ls /nope
ls: cannot access '/nope': No such file or directory
learner@embeddex:~/lab$ # the shell's own descriptors 0, 1 and 2 all point at the terminal
learner@embeddex:~/lab$ ls -l /proc/$$/fd
total 0
lr-x------ 1 learner learner 64 Nov 15 05:45 0 -> pipe:[7077]
l-wx------ 1 learner learner 64 Nov 15 05:45 1 -> /tmp/embx-out.nPRi98
l-wx------ 1 learner learner 64 Nov 15 05:45 10 -> pipe:[7078]
l-wx------ 1 learner learner 64 Nov 15 05:45 11 -> pipe:[7079]
l-wx------ 1 learner learner 64 Nov 15 05:45 2 -> /tmp/embx-out.nPRi98
lrwx------ 1 learner learner 64 Nov 15 05:45 7 -> /dev/ptmx
02

Redirection: where output goes

3 min read

Redirection is how you move a stream off the terminal. The operators are short, and each one moves exactly one stream.

  • > sends stdout to a file, creating it if needed and truncating it to zero bytes first.
  • >> sends stdout to a file but appends, so existing content survives.
  • < makes stdin come from a file instead of the keyboard.
  • 2> sends stderr to a file, leaving stdout on the terminal.
  • 2>&1 makes stderr go wherever stdout currently points.
  • &> sends both stdout and stderr to the same file; it is shorthand for > file 2>&1.
bash
echo hello > greeting.txt     # create or truncate, then write
echo world >> greeting.txt    # append a second line
sort < greeting.txt           # feed the file into sort's stdin
ls /nope 2> errors.txt        # capture only the error
ls /nope &> all.txt           # capture both streams

The difference between > and >> is the one that destroys data. > opens the file with the truncate flag, wiping it before the first byte is written. >> opens it in append mode, so every write lands at the end. Reaching for the wrong one is the classic way to empty a log you meant to extend.

> wipes the file first

> truncates before writing. If you meant to add a line and typed >, the old content is gone the instant the shell opens the file. Use >> to append.

Two extras round out the toolkit. set -o noclobber makes > refuse to overwrite an existing file, and >| overrides that guard when you really do want to clobber. The special file /dev/null is the bit bucket: writing to it discards the bytes, which is how you silence noise on a busy or embedded system where every byte of log storage counts.

bash
set -o noclobber
echo x > f          # refuses if f already exists
echo x >| f         # forces the overwrite
noisy_tool 2>/dev/null

2>/dev/null is the everyday form for throwing away error noise: the same bit bucket, aimed at stderr only. Two more forms feed input without a file. A here-document sends several lines into a command's stdin, and a here-string sends a single string as one line:

bash
grep error <<'EOF'
line one
an error line
EOF
wc -l <<< 'a single line'

The marker after << ends the here-document, and quoting it stops the shell from expanding $ and backticks inside the block. These matter less in interactive work and more inside scripts, where a command's whole input can sit inline rather than in a separate file.

Redirection operators>stdoutfile(truncate)>>stdoutfile(append)<filestdin2>stderrfile2>&1stderrwhereverstdoutgoes&>stdout +stderrsame fileeach operator moves exactly one stream
The redirection operators and which stream each one moves, and to where.
Redirecting stdout and stderr
learner@embeddex:~/lab$ # > truncates the file, then writes one line
learner@embeddex:~/lab$ echo hello > greeting.txt
learner@embeddex:~/lab$ # >> appends instead of wiping
learner@embeddex:~/lab$ echo world >> greeting.txt
learner@embeddex:~/lab$ # both lines survived
learner@embeddex:~/lab$ cat greeting.txt
hello
world
learner@embeddex:~/lab$ # the error disappears from the screen
learner@embeddex:~/lab$ ls /nope 2> errors.txt
learner@embeddex:~/lab$ # because 2> sent stderr into the file
learner@embeddex:~/lab$ cat errors.txt
ls: cannot access '/nope': No such file or directory
learner@embeddex:~/lab$ # &> captures stdout and stderr together
learner@embeddex:~/lab$ ls /nope &> all.txt
learner@embeddex:~/lab$ # the error landed here too
learner@embeddex:~/lab$ cat all.txt
ls: cannot access '/nope': No such file or directory
03

Redirection order matters

4 min read

2>&1 does not mean send both streams to the same place as a permanent rule. It means: make file descriptor 2 point at whatever file descriptor 1 points at this exact moment. Because redirections are evaluated left to right, the two common spellings diverge.

cmd > f 2>&1 is the correct one. First > f makes fd 1 point at the file f. Then 2>&1 makes fd 2 point where fd 1 points now, which is f. Both streams land in the file.

cmd 2>&1 > f is the trap. First 2>&1 makes fd 2 point where fd 1 points now, which is still the terminal. Then > f moves fd 1 to the file. stderr keeps pointing at the terminal, and the file receives only stdout. For a command that only produces an error, the file ends up empty while the error still prints on screen.

Step by stepEvaluating cmd > f 2>&1 left to right
  1. Open f: the shell opens or creates f, truncating it, and puts it on fd 1.

  2. stdout moved: fd 1 no longer points at the terminal; it points at f.

  3. Duplicate fd 1: 2>&1 copies fd 1's target onto fd 2.

  4. stderr follows: fd 2 now points at f too, because fd 1 already did.

  5. Both streams in f: stdout and stderr write into the same file.

The reverse order is one step swapped: duplicate first, then move stdout. stderr copies the terminal because that is where stdout still pointed at the time. Reproduce it in seconds with a nonexistent path: ls /nope 2>&1 > f; cat f prints the error to the screen and leaves f empty, while ls /nope > f 2>&1; cat f puts the error in f. Run both and watch where the message lands.

One portability note: &> is a bash convenience. POSIX sh spells the same thing > f 2>&1, so a script that starts with #!/bin/sh should use the long form to stay portable across shells and minimal BusyBox environments.

Going deeperWhat the shell does underneath: open, dup2, exec

The shell implements all of this with three system calls. > calls open(path, O_WRONLY|O_CREAT|O_TRUNC) and puts the returned descriptor on fd 1 via dup2. 2>&1 is a plain dup2(1, 2): copy fd 1's entry onto fd 2. After all redirections, the shell calls execve to replace itself with the command; file descriptors survive exec by design, which is exactly how redirection reaches the child. A shell marks its own pipe and redirect descriptors close-on-exec so the command does not inherit clutter. The order you type the operators is, in effect, the order of open and dup2 calls.

The file-empty trap

If you capture an error with cmd 2>&1 > f and find f empty, it is not the command's fault. stderr went to the terminal because the duplicate ran before stdout moved. Write > f 2>&1.

Redirection ordercmd 2>&1 > fcmd > f 2>&1stderr goes tothe terminalfstdout goes tofff ends up withstdout onlystdout andstderr
How the two orderings 2>&1 > f and > f 2>&1 send stderr to different places.
Why 2>&1 order changes everything
learner@embeddex:~/lab$ # trap: stderr copies the terminal before stdout moves to the file
learner@embeddex:~/lab$ ls /nope 2>&1 > f
ls: cannot access '/nope': No such file or directory
learner@embeddex:~/lab$ # the file is empty: the error never went in
learner@embeddex:~/lab$ wc -c f
0 f
learner@embeddex:~/lab$ # fixed: stdout moves first, then stderr follows it
learner@embeddex:~/lab$ ls /nope > f 2>&1
learner@embeddex:~/lab$ # now the error is captured
learner@embeddex:~/lab$ cat f
ls: cannot access '/nope': No such file or directory
04

Pipes and tee

3 min read

The pipe character | joins two commands so the bytes one writes become the bytes the next reads. No file is involved, and the two commands run at the same time, with the kernel moving data between them.

bash
ls -l /usr/bin | wc -l       # count the entries
dmesg | tail -n 20           # show only the last boot messages

tee sits inside a pipeline and splits the stream. It writes a copy to the file you name and passes the same bytes onward to the next command. Add -a to append rather than overwrite.

bash
cmd | tee capture.txt | wc -l

The chain above saves a full copy to capture.txt while still counting lines. On a board where you are debugging serial output, tee lets you watch the stream live and keep a log at the same time.

The shell builds a pipe in three steps. It calls pipe() to create a kernel buffer with a read end and a write end. It forks, giving each side its own copy of the descriptors. The left child points fd 1 at the write end and runs the first command; the right child points fd 0 at the read end and runs the second. The descriptors the children do not need are closed, so the second command sees end-of-file when the first finishes.

The exit status of a pipeline is the status of its last command, but bash keeps every stage's status in the PIPESTATUS array. After false | true, the pipeline as a whole succeeds because true ran last, while PIPESTATUS[0] is 1. A script that must know a producer failed even though the consumer completed reads that array.

bash
false | true
echo "${PIPESTATUS[0]}"    # 1, the status of false

A pipeline starts both commands at once, unlike writing to a temporary file and reading it back, which forces the first command to finish before the second starts. Because the pipe is a fixed buffer, a producer that outruns its consumer is slowed rather than allowed to fill the disk, and no intermediate file ever needs to exist. That combination of concurrency and bounded memory is why pipelines are the default way to move data between programs on Linux.

One purpose per tool

A pipe is an assembly line of single-purpose tools. sort does not need to know about grep; each reads stdin and writes stdout, and the pipe is what glues them into a bigger machine.

A pipe with teecmd awritesstdoutpipekernelbufferteealso writesfilecmd breads stdincmd awritesstdoutpipekernelbufferteealso writesfilecmd breads stdin
A pipe connecting two commands, with tee splitting a copy off to a file.
Pipes and tee
learner@embeddex:~/lab$ # ls's stdout feeds wc's stdin: count the entries
learner@embeddex:~/lab$ ls -l /usr/bin | wc -l
1035
learner@embeddex:~/lab$ # tee saves a copy to the file and passes the stream on
learner@embeddex:~/lab$ printf 'a\nb\nc\n' | tee capture.txt | wc -l
3
learner@embeddex:~/lab$ # the copy tee kept while the count flowed onward
learner@embeddex:~/lab$ cat capture.txt
a
b
c
learner@embeddex:~/lab$ # the pipeline reports true's success, but PIPESTATUS[0] still holds fal
se's failure
learner@embeddex:~/lab$ false | true; echo "${PIPESTATUS[0]}"
1
05

grep and basic regular expressions

5 min read

grep reads lines and prints the ones that match a pattern. The name stands for global regular expression print, and the pattern is a regular expression, not a plain word search.

The core options cover most log work. -c counts matching lines instead of printing them, -v prints lines that do not match, -i ignores case, -w requires the match to be a whole word, and -n prints line numbers.

bash
grep error app.log        # lines containing 'error' anywhere
grep -c error app.log     # how many such lines
grep -v error app.log     # lines without 'error'
grep -i error app.log     # ignore case
grep -w error app.log     # the whole word 'error' only
grep -n error app.log     # with line numbers

Three more options serve files rather than a single log. -r searches a directory recursively, -l prints only the names of files that contain a match, and -o prints just the matching part of each line, not the whole line.

bash
grep -r 'TODO' src/         # search every file under src/
grep -rl 'TODO' src/        # only the filenames that match
grep -o 'error[0-9]*' app.log   # only the error codes

The basic regular expression operators give you a real pattern language. . matches any single character, ^ anchors to the start of the line, $ anchors to the end, [abc] matches any one listed character, [a-z] a range, [^a-z] anything not in the range, and * repeats the previous thing zero or more times. A . never crosses a line boundary, so grep 'start.*end' cannot span two lines; every line is matched on its own. To insist on a whole line, anchor both ends, as in grep '^[A-Z].*[0-9]$', which selects lines that begin with a capital letter and end with a digit.

bash
grep '^ERROR' app.log            # lines starting with ERROR
grep '\.conf$' files.txt         # lines ending in .conf
grep '^[0-9]' data.txt           # lines starting with a digit
grep 'colou\{0,1\}r' text.txt    # color or colour

Notice the backslashes in the last line. In the basic dialect, \{, \}, \(, \) and \| are the special forms, while the bare characters are literal. You will meet the cleaner extended dialect shortly.

Play with the pattern below:

grep 'error' matches 'error' anywhere in a line, so it also matches inside 'noerror', and it is case-sensitive, so 'ERROR: disk full' does not match. Add the i flag for the grep -i effect; anchoring to a whole word gives the grep -w behaviour.

Stop early on huge logs

grep -m 5 pattern prints five matches and stops reading, which is useful on a slow or embedded board scanning a large file. For a literal string with no regex meaning, grep -F is faster and matches the text exactly.

Going deeperHow grep matches: the engine

GNU grep compiles a basic or extended regex into a lazy deterministic finite automaton and simulates it in linear time, reporting the leftmost-longest match, as POSIX requires. That is why a plain grep never blows up the way a backtracking engine can on a bad pattern. For fixed strings, grep -F skips the automaton and uses an Aho-Corasick machine, or Boyer-Moore for a single literal, which is far faster. The backtracking approach, used by PCRE (grep -P) and many programming languages, tries one path and unwinds on failure, which on crafted patterns can take exponential time. This distinction is the subject of Russ Cox's paper in the references.

grep filtering linesread a linepatternmatch?regexmatch:print lineExit from step 2: no matchdiscard line↻ Repeat from step 1read a linepatternmatch?regexExit from step 2: no matchdiscard linematch:print line↻ Repeat from step 1
grep reading lines, matching each against the pattern, and printing or discarding it.
grep matches patterns, not words
learner@embeddex:~/lab$ # grep matches the pattern anywhere, so noerror matches too
learner@embeddex:~/lab$ grep error app.log
an error occurred
noerror here
learner@embeddex:~/lab$ # -w requires a whole word, so noerror is left out
learner@embeddex:~/lab$ grep -w error app.log
an error occurred
learner@embeddex:~/lab$ # -i ignores case, so ERROR matches too
learner@embeddex:~/lab$ grep -i error app.log
ERROR: disk full
an error occurred
noerror here
learner@embeddex:~/lab$ # -c counts matching lines instead of printing them
learner@embeddex:~/lab$ grep -c error app.log
2
learner@embeddex:~/lab$ # the ^ anchor matches only lines that start with ERROR
learner@embeddex:~/lab$ grep '^ERROR' app.log
ERROR: disk full
06

Inside the pipe

4 min read

A pipe is not a file and not a queue that grows forever. The kernel allocates a circular buffer in memory and hands the process two descriptors, one for each end. On Linux the default capacity is 16 pages, or 64 KiB, and the constant PIPE_BUF sets the size, 4096 bytes, under which a write is guaranteed atomic.

The capacity is what makes pipelines safe on small machines. A fast producer filling the buffer blocks in write until the reader drains some bytes; a reader on an empty buffer blocks in read until the writer produces. The two commands run concurrently, and the pipe enforces backpressure, so a pipeline never buffers an entire file in memory.

The other edge case is a vanished reader. If the reading process exits and the writer keeps writing, the kernel delivers SIGPIPE, whose default action terminates the writer. The classic demonstration is yes, which writes forever:

bash
yes | head -n 1
echo "${PIPESTATUS[0]}"    # 141 in bash

head prints one line and exits, closing the read end. yes's next write hits a pipe with no reader, the kernel raises SIGPIPE, and yes dies with status 141, which is 128 plus the signal number 13.

A process can choose to ignore SIGPIPE and keep writing. Then write returns an error with errno set to EPIPE, which is why some programs print write error: Broken pipe instead of dying. Command-line tools mostly leave the default and simply stop, which is the behaviour you want when the reader has had enough and the rest of the output is useless.

The fixed size is a trade-off. A small buffer means a writer blocks sooner and the reader wakes sooner, giving low latency for interactive use; a large buffer lets a producer run ahead and amortises the cost of waking the consumer, giving higher throughput. Linux picked 64 KiB as a middle ground, and the size is tunable per pipe for workloads that need more.

Going deeperAtomic writes and resizing the buffer

Writes of PIPE_BUF bytes or fewer are atomic: interleaved writers cannot tear each other's data, so concurrent producers writing short records stay intact. Writes larger than PIPE_BUF may interleave with others. The capacity can be changed per pipe with fcntl(fd, F_SETPIPE_SZ, n), bounded by /proc/sys/fs/pipe-max-size; privileged processes may raise it beyond the default 64 KiB for high-throughput pipelines. The details live in the pipe(7) man page.

Inside a pipewriterkernel pipereaderwrite 4 KBaccepted, bufferfillsread4 KB outwrite, buffer fullwriter blocksreader exitsSIGPIPE1. writer → kernel pipewrite 4 KB2. kernel pipe → writeraccepted, buffer fills3. reader → kernel piperead4. kernel pipe → reader4 KB out5. writer → kernel pipewrite, buffer full6. kernel pipe → writerwriter blocks7. reader → kernel pipereader exits8. kernel pipe → writerSIGPIPE
A sequence diagram of a writer, the kernel pipe buffer and a reader, including blocking and SIGPIPE.
07

Buffering: why stdout lags stderr

3 min read

Two layers sit between your program and the bytes on disk. The kernel moves data through file descriptors directly, but the C standard library adds its own buffer on top, and that buffer is the source of surprising ordering.

The library follows three rules. stdout is line buffered when it points at a terminal, meaning it flushes on every newline. stdout is fully buffered when it points at a pipe or a file, meaning it flushes only when its buffer fills or when the program exits. stderr is never buffered, so its bytes reach the device immediately.

The consequence shows up whenever you pipe a program that prints to both streams. stdout's output sits in the buffer while stderr's races straight through, so the error can appear before the result it belongs to:

bash
python3 -c "import sys; print('out'); print('err', file=sys.stderr)" 2>&1 | cat

Typical output (yours will differ) shows err before out, because the piped stdout was fully buffered while stderr flushed at once. On the terminal, with stdout line buffered, they appear in the written order. The same program, the same lines, a different order.

You can change the buffering without editing the program. stdbuf -oL forces line buffering, stdbuf -o0 forces no buffering, and python3 -u does the same for Python.

bash
stdbuf -oL cmd 2>&1 | cat

Run the earlier Python one-liner again with python3 -u (or set PYTHONUNBUFFERED=1) and the two lines come out in the written order even through the pipe. Note that stdbuf would not help here: it only changes the C library's stdio buffering, so it works for programs such as grep, cut and awk, while Python manages its own output buffers. The program did not change; only its buffering policy did, which is the point of the fix.

Going deeperWhy stderr is unbuffered by design

The C library sets stderr unbuffered so a diagnostic is seen even when the program crashes before flushing, since a buffered error could be lost with the process. The setvbuf call lets a program choose line, full or no buffering per stream, and stdbuf achieves the same from outside by preloading a library that overrides those choices. The reason the terminal gets line buffering while a pipe gets full buffering is an old performance trade-off: interactive output should feel immediate, while batch output benefits from fewer, larger writes.

Stream bufferingstdout toterminalstdout topipe/filestderrbufferingline bufferedfullybufferedunbufferedflushed whennewlinebuffer fullor exitimmediatelyso errors appearon each linelatefirst
How stdout and stderr are buffered differently on a terminal versus through a pipe.
08

Extended regular expressions

2 min read

The basic dialect forces a backslash before \|, \+, \?, \( and \{. The extended dialect, enabled with grep -E (or the old name egrep), makes those characters operators directly, which reads far better for real patterns.

bash
grep -E 'error|warn|fatal' app.log        # alternation
grep -E 'https?://' urls.txt              # s is optional
grep -E 'colou?r' text.txt                # color or colour
grep -E '[0-9]{4}-[0-9]{2}-[0-9]{2}' log  # a date
grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' log # an IPv4 shape

Each operator deserves a line. | means or, matching if either side fits. ? makes the previous item optional, + means one or more, and * means zero or more. {n,m} repeats the previous item between n and m times. Parentheses group items so an operator can apply to the whole group, as in the repeated octet pattern above. Anchors ^ and $ and classes like [0-9] work the same in both dialects.

One thing that trips people switching from other languages: POSIX extended regular expressions have no \d. The portable spelling for a digit is [0-9] or the POSIX class [[:digit:]], alongside [[:alpha:]], [[:alnum:]] and [[:space:]].

Try the extended operators below:

grep -E 'error|warn|fatal' prints the warn, error and fatal lines because | means or in the extended dialect; in basic grep the same | is literal, so the pattern matches nothing.

Both dialects match greedily: they take the longest string the pattern allows, from the leftmost position where a match can start. There is no lazy or non-greedy operator in POSIX regular expressions, so there is no way to ask for the shortest match; that idea belongs to Perl-style engines, which GNU grep offers only through -P. Anchoring both ends turns a loose search into an exact check, so grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}$' accepts nothing but a bare IPv4 address with no surrounding text.

BusyBox grep is a subset

On an embedded root filesystem, grep is usually the BusyBox applet. It supports -E and -F but not -P, and its extended dialect is a subset of GNU's. Test a pattern on the target board, not just on your workstation, because an option that works in GNU grep may be absent on the device.

Basic vs extended regexBRE (grep)ERE (grep -E)alternation\||grouping\( \)( )plus andoptional\+ \?+ ?repeat count\{n,m\}{n,m}
The same operators spelled in the basic and extended dialects side by side.
09

The text toolkit and the full pipeline

4 min read

cut extracts columns. -d sets the delimiter and -f selects fields, while -c selects character positions.

bash
cut -d: -f1 /etc/passwd      # every username
cut -c1-10 access.log        # the first ten characters

sort orders lines, with -n for numeric order, -r to reverse and -k to pick a key. uniq collapses adjacent duplicates, and -c prefixes each with a count. Because uniq only compares neighbours, you sort first. sort -u removes duplicates outright during sorting, a shortcut when you only want distinct values; reach for uniq -c when the count itself is the point.

bash
sort words.txt | uniq -c     # count each distinct word
sort -rn sizes.txt           # largest first

tr translates, deletes or squeezes characters. It maps characters positionally, so tr 'A-Z' 'a-z' lowercases.

bash
echo 'HELLO' | tr 'A-Z' 'a-z'      # hello
tr -d '\r' < windows.txt           # strip carriage returns
tr -s ' ' < messy.txt              # collapse runs of spaces

xargs turns lines of stdin into arguments for another command. It is how a pipeline feeds a command that reads arguments, not stdin, and it splits the list into batches that fit within the kernel's command-line length limit.

bash
grep -l 'TODO' *.c | xargs grep -n 'TODO'
find . -name '*.log' -print0 | xargs -0 grep -l error

-n bounds how many arguments each invocation receives, and -I{} substitutes a name at a fixed position, which matters when the argument must appear twice:

bash
seq 1 20 | xargs -n 5 echo         # five arguments per echo
echo report | xargs -I{} cp {} {}.bak

The kernel caps the size of a single command line, reported by getconf ARG_MAX. xargs respects it by splitting the input into batches, which is why grep TODO *.c over tens of thousands of files fails with argument list too long. Note that the glob is expanded by the shell before any command runs, so you cannot rescue it by piping its result into xargs; generate the names without one giant command line instead, with find . -name '*.c' -print0 | xargs -0 grep TODO or the bash builtin printf '%s' *.c | xargs -0 grep TODO (a builtin is not subject to the limit).

The -print0 and -0 pair uses a NUL byte as the separator, which survives filenames containing spaces. Now the pieces combine into the log-analysis pipeline.

1extract one field

cut -d' ' -f1 access.log takes the first space-separated field of each line, the client IP.

2group the duplicates

sort puts identical IPs next to each other, so uniq -c can count them in one pass.

3rank and trim

sort -rn orders by count descending, and head keeps the top few.

bash
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head
grep -c ' 404 ' access.log

Watch each stage transform the sample below:

cut -d' ' -f1 extracts each client IP, sort makes duplicates adjacent, uniq collapses them, and head keeps the first three lines. The real top-IP ranking adds uniq -c to count and sort -rn to order by count, which this simulation does not implement.

The spaces around 404 stop the pattern matching 1404 or 4040. A real access log has spaces inside the quoted request line, which is why cut on a later field produces empty columns; field extraction that ignores whitespace runs is the job of awk, which the next lesson covers.

Streaming beats loading on small boards

A board with 512 MB of RAM cannot slurp a multi-gigabyte log. The pipeline above streams line by line: only sort spills to disk when the input outgrows memory. Use sort -S to bound its memory or sort -T to point its temporary files at larger storage.

The same mistakes recur, and each has a symptom, a cause and a fix.

Symptom Cause Fix
Error still prints after cmd > f > redirects stdout only cmd > f 2>&1 or &> f
f is empty after cmd 2>&1 > f duplicate ran before stdout moved cmd > f 2>&1
> f wiped a file you meant to extend > truncates use >>
uniq -c splits one value into several duplicates were not adjacent sort | uniq -c
grep word matches noerror too grep matches patterns, not words grep -w word
grep 'a|b' matches nothing | is alternation only in -E grep -E 'a|b'
cut -d' ' -f2 gives empty fields every space is a delimiter use awk, or tr -s ' ' first
The log-analysis pipelinecut -f1IP columnsortgroupduplicatesuniq -ccount eachsort -rnrank bycountheadtop Ncut -f1IP columnsortgroupduplicatesuniq -ccount eachsort -rnrank bycountheadtop N
The full log pipeline: cut, sort, uniq -c, sort -rn and head.
Rank the top client IPs
192.0.2.99
198.51.100.23
203.0.113.50
203.0.113.7
203.0.113.7
203.0.113.7
learner@embeddex:~/lab$ # uniq -c counts each run of adjacent duplicates
learner@embeddex:~/lab$ cut -d' ' -f1 access.log | sort | uniq -c
      1 192.0.2.44
      1 192.0.2.99
      1 198.51.100.23
      1 203.0.113.50
      3 203.0.113.7
learner@embeddex:~/lab$ # sort -rn ranks by count, head keeps the top
learner@embeddex:~/lab$ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head
      3 203.0.113.7
      1 203.0.113.50
      1 198.51.100.23
      1 192.0.2.99
      1 192.0.2.44
learner@embeddex:~/lab$ # the spaces around 404 stop it matching 1404 or 4040
learner@embeddex:~/lab$ grep -c ' 404 ' access.log
1
10

Try it

1 min read

Open the lab for this session and run it in a scratch directory. Generate a small access log, then build the reports from this lesson: the top client IPs with cut -d' ' -f1 | sort | uniq -c | sort -rn | head, the 404 count with grep -c ' 404 ', and a redirection exercise that captures stdout and stderr separately with > f 2>&1. Then run the same pipeline over SSH on your Raspberry Pi and compare the BusyBox tools' options with the GNU versions. The practice pack adds predict-the-output drills for redirection order and buffering.

Key takeaways

  • Every process gets three streams: stdin (0), stdout (1) and stderr (2).
  • > truncates and >> appends; 2>&1 and &> capture stderr together with stdout.
  • Redirections are applied left to right, so 2>&1 >file and >file 2>&1 mean different things.
  • A pipe connects one process's stdout to the next's stdin; tee splits the stream to a file and onward.
  • grep matches regular expressions anywhere in a line, not whole words; -w and -E change that.
  • sort before uniq -c, because uniq only collapses adjacent identical lines.
  • A pipe is a fixed-size kernel buffer that blocks the writer when full and delivers SIGPIPE when the reader is gone.
  • stdout is line-buffered on a terminal but fully buffered in a pipe; stderr is never buffered.

Reflect

  1. Explain to a colleague, in two sentences: redirect stdout, stderr and stdin independently with >, >>, <, 2> and &>, and predict where each stream lands.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close