Three streams and file descriptors
3 min readEvery process on Linux starts life with three open channels. They are called the standard streams, and they are referred to by number, not by name. Standard input is file descriptor 0, standard output is file descriptor 1, and standard error is file descriptor 2.
a small integer a process uses to refer to an open file, pipe or device
the three channels every process opens at startup: stdin (0), stdout (1) and stderr (2)
By default all three point at your terminal. When you type into bash, the shell reads from stdin. When a command prints a result, it writes to stdout. When it reports a problem, it writes to stderr. The two output streams look identical on screen, but they are separate channels, and that separation is the whole point of this lesson.
ls /etc/passwd # writes the path to stdout
ls /nope # writes an error to stderr
Both lines appear in your terminal, which hides the difference. Typical output (yours will differ) looks like two normal lines, but the first came from fd 1 and the second from fd 2. You want that split because you usually save results to a file while keeping errors visible in front of you.
You can inspect a running process's descriptors through /proc. The shell's own descriptors show up under /proc/$$/fd:
ls -l /proc/$$/fd
Every tool from cat to a long-running daemon follows the same three-descriptor convention. A headless board, with no one watching a screen, depends on it: a service logs results to one place and faults to another, and the two never mix.
Why two output streams at all? A single stream cannot separate an answer from a warning about how the answer was produced. Unix has kept results and diagnostics apart since the 1970s for exactly this reason, and the numbering 0, 1 and 2 is fixed by convention, not by hardware. The terminal is itself a file, usually /dev/tty, and all three descriptors simply point at it until you redirect them. On a board with no monitor they point instead at a serial console such as /dev/ttyAMA0 or /dev/ttyS0, which is why the serial connection you used to reach your Raspberry Pi shows the same streams.
learner@embeddex:~/lab$ # a successful lookup writes the path to stdout learner@embeddex:~/lab$ ls /etc/passwd /etc/passwd learner@embeddex:~/lab$ # a failure writes the error to stderr, not stdout learner@embeddex:~/lab$ ls /nope ls: cannot access '/nope': No such file or directory learner@embeddex:~/lab$ # the shell's own descriptors 0, 1 and 2 all point at the terminal learner@embeddex:~/lab$ ls -l /proc/$$/fd total 0 lr-x------ 1 learner learner 64 Nov 15 05:45 0 -> pipe:[7077] l-wx------ 1 learner learner 64 Nov 15 05:45 1 -> /tmp/embx-out.nPRi98 l-wx------ 1 learner learner 64 Nov 15 05:45 10 -> pipe:[7078] l-wx------ 1 learner learner 64 Nov 15 05:45 11 -> pipe:[7079] l-wx------ 1 learner learner 64 Nov 15 05:45 2 -> /tmp/embx-out.nPRi98 lrwx------ 1 learner learner 64 Nov 15 05:45 7 -> /dev/ptmx
Redirection: where output goes
3 min readchanging which file or device one of the three standard streams points to
Redirection is how you move a stream off the terminal. The operators are short, and each one moves exactly one stream.
>sends stdout to a file, creating it if needed and truncating it to zero bytes first.>>sends stdout to a file but appends, so existing content survives.<makes stdin come from a file instead of the keyboard.2>sends stderr to a file, leaving stdout on the terminal.2>&1makes stderr go wherever stdout currently points.&>sends both stdout and stderr to the same file; it is shorthand for> file 2>&1.
echo hello > greeting.txt # create or truncate, then write
echo world >> greeting.txt # append a second line
sort < greeting.txt # feed the file into sort's stdin
ls /nope 2> errors.txt # capture only the error
ls /nope &> all.txt # capture both streams
The difference between > and >> is the one that destroys data. > opens the file with the truncate flag, wiping it before the first byte is written. >> opens it in append mode, so every write lands at the end. Reaching for the wrong one is the classic way to empty a log you meant to extend.
> truncates before writing. If you meant to add a line and typed >, the old content is gone the instant the shell opens the file. Use >> to append.
Two extras round out the toolkit. set -o noclobber makes > refuse to overwrite an existing file, and >| overrides that guard when you really do want to clobber. The special file /dev/null is the bit bucket: writing to it discards the bytes, which is how you silence noise on a busy or embedded system where every byte of log storage counts.
set -o noclobber
echo x > f # refuses if f already exists
echo x >| f # forces the overwrite
noisy_tool 2>/dev/null
2>/dev/null is the everyday form for throwing away error noise: the same bit bucket, aimed at stderr only. Two more forms feed input without a file. A here-document sends several lines into a command's stdin, and a here-string sends a single string as one line:
grep error <<'EOF'
line one
an error line
EOF
wc -l <<< 'a single line'
The marker after << ends the here-document, and quoting it stops the shell from expanding $ and backticks inside the block. These matter less in interactive work and more inside scripts, where a command's whole input can sit inline rather than in a separate file.
learner@embeddex:~/lab$ # > truncates the file, then writes one line learner@embeddex:~/lab$ echo hello > greeting.txt learner@embeddex:~/lab$ # >> appends instead of wiping learner@embeddex:~/lab$ echo world >> greeting.txt learner@embeddex:~/lab$ # both lines survived learner@embeddex:~/lab$ cat greeting.txt hello world learner@embeddex:~/lab$ # the error disappears from the screen learner@embeddex:~/lab$ ls /nope 2> errors.txt learner@embeddex:~/lab$ # because 2> sent stderr into the file learner@embeddex:~/lab$ cat errors.txt ls: cannot access '/nope': No such file or directory learner@embeddex:~/lab$ # &> captures stdout and stderr together learner@embeddex:~/lab$ ls /nope &> all.txt learner@embeddex:~/lab$ # the error landed here too learner@embeddex:~/lab$ cat all.txt ls: cannot access '/nope': No such file or directory
Redirection order matters
4 min read2>&1 does not mean send both streams to the same place as a permanent rule. It means: make file descriptor 2 point at whatever file descriptor 1 points at this exact moment. Because redirections are evaluated left to right, the two common spellings diverge.
cmd > f 2>&1 is the correct one. First > f makes fd 1 point at the file f. Then 2>&1 makes fd 2 point where fd 1 points now, which is f. Both streams land in the file.
cmd 2>&1 > f is the trap. First 2>&1 makes fd 2 point where fd 1 points now, which is still the terminal. Then > f moves fd 1 to the file. stderr keeps pointing at the terminal, and the file receives only stdout. For a command that only produces an error, the file ends up empty while the error still prints on screen.
Open f: the shell opens or creates
f, truncating it, and puts it on fd 1.stdout moved: fd 1 no longer points at the terminal; it points at
f.Duplicate fd 1:
2>&1copies fd 1's target onto fd 2.stderr follows: fd 2 now points at
ftoo, because fd 1 already did.Both streams in f: stdout and stderr write into the same file.
The reverse order is one step swapped: duplicate first, then move stdout. stderr copies the terminal because that is where stdout still pointed at the time. Reproduce it in seconds with a nonexistent path: ls /nope 2>&1 > f; cat f prints the error to the screen and leaves f empty, while ls /nope > f 2>&1; cat f puts the error in f. Run both and watch where the message lands.
One portability note: &> is a bash convenience. POSIX sh spells the same thing > f 2>&1, so a script that starts with #!/bin/sh should use the long form to stay portable across shells and minimal BusyBox environments.
Going deeperWhat the shell does underneath: open, dup2, exec
The shell implements all of this with three system calls. > calls open(path, O_WRONLY|O_CREAT|O_TRUNC) and puts the returned descriptor on fd 1 via dup2. 2>&1 is a plain dup2(1, 2): copy fd 1's entry onto fd 2. After all redirections, the shell calls execve to replace itself with the command; file descriptors survive exec by design, which is exactly how redirection reaches the child. A shell marks its own pipe and redirect descriptors close-on-exec so the command does not inherit clutter. The order you type the operators is, in effect, the order of open and dup2 calls.
If you capture an error with cmd 2>&1 > f and find f empty, it is not the command's fault. stderr went to the terminal because the duplicate ran before stdout moved. Write > f 2>&1.
learner@embeddex:~/lab$ # trap: stderr copies the terminal before stdout moves to the file learner@embeddex:~/lab$ ls /nope 2>&1 > f ls: cannot access '/nope': No such file or directory learner@embeddex:~/lab$ # the file is empty: the error never went in learner@embeddex:~/lab$ wc -c f 0 f learner@embeddex:~/lab$ # fixed: stdout moves first, then stderr follows it learner@embeddex:~/lab$ ls /nope > f 2>&1 learner@embeddex:~/lab$ # now the error is captured learner@embeddex:~/lab$ cat f ls: cannot access '/nope': No such file or directory
Pipes and tee
3 min reada kernel buffer that connects one process's stdout to another process's stdin
The pipe character | joins two commands so the bytes one writes become the bytes the next reads. No file is involved, and the two commands run at the same time, with the kernel moving data between them.
ls -l /usr/bin | wc -l # count the entries
dmesg | tail -n 20 # show only the last boot messages
tee sits inside a pipeline and splits the stream. It writes a copy to the file you name and passes the same bytes onward to the next command. Add -a to append rather than overwrite.
cmd | tee capture.txt | wc -l
The chain above saves a full copy to capture.txt while still counting lines. On a board where you are debugging serial output, tee lets you watch the stream live and keep a log at the same time.
The shell builds a pipe in three steps. It calls pipe() to create a kernel buffer with a read end and a write end. It forks, giving each side its own copy of the descriptors. The left child points fd 1 at the write end and runs the first command; the right child points fd 0 at the read end and runs the second. The descriptors the children do not need are closed, so the second command sees end-of-file when the first finishes.
The exit status of a pipeline is the status of its last command, but bash keeps every stage's status in the PIPESTATUS array. After false | true, the pipeline as a whole succeeds because true ran last, while PIPESTATUS[0] is 1. A script that must know a producer failed even though the consumer completed reads that array.
false | true
echo "${PIPESTATUS[0]}" # 1, the status of false
A pipeline starts both commands at once, unlike writing to a temporary file and reading it back, which forces the first command to finish before the second starts. Because the pipe is a fixed buffer, a producer that outruns its consumer is slowed rather than allowed to fill the disk, and no intermediate file ever needs to exist. That combination of concurrency and bounded memory is why pipelines are the default way to move data between programs on Linux.
A pipe is an assembly line of single-purpose tools. sort does not need to know about grep; each reads stdin and writes stdout, and the pipe is what glues them into a bigger machine.
learner@embeddex:~/lab$ # ls's stdout feeds wc's stdin: count the entries
learner@embeddex:~/lab$ ls -l /usr/bin | wc -l
1035
learner@embeddex:~/lab$ # tee saves a copy to the file and passes the stream on
learner@embeddex:~/lab$ printf 'a\nb\nc\n' | tee capture.txt | wc -l
3
learner@embeddex:~/lab$ # the copy tee kept while the count flowed onward
learner@embeddex:~/lab$ cat capture.txt
a
b
c
learner@embeddex:~/lab$ # the pipeline reports true's success, but PIPESTATUS[0] still holds fal
se's failure
learner@embeddex:~/lab$ false | true; echo "${PIPESTATUS[0]}"
1grep and basic regular expressions
5 min readgrep reads lines and prints the ones that match a pattern. The name stands for global regular expression print, and the pattern is a regular expression, not a plain word search.
a pattern language that describes a set of strings, using characters plus operators such as anchors, classes and quantifiers
the default regex dialect of grep, in which the operators +, ?, |, ( and { need a backslash to be special
The core options cover most log work. -c counts matching lines instead of printing them, -v prints lines that do not match, -i ignores case, -w requires the match to be a whole word, and -n prints line numbers.
grep error app.log # lines containing 'error' anywhere
grep -c error app.log # how many such lines
grep -v error app.log # lines without 'error'
grep -i error app.log # ignore case
grep -w error app.log # the whole word 'error' only
grep -n error app.log # with line numbers
Three more options serve files rather than a single log. -r searches a directory recursively, -l prints only the names of files that contain a match, and -o prints just the matching part of each line, not the whole line.
grep -r 'TODO' src/ # search every file under src/
grep -rl 'TODO' src/ # only the filenames that match
grep -o 'error[0-9]*' app.log # only the error codes
The basic regular expression operators give you a real pattern language. . matches any single character, ^ anchors to the start of the line, $ anchors to the end, [abc] matches any one listed character, [a-z] a range, [^a-z] anything not in the range, and * repeats the previous thing zero or more times. A . never crosses a line boundary, so grep 'start.*end' cannot span two lines; every line is matched on its own. To insist on a whole line, anchor both ends, as in grep '^[A-Z].*[0-9]$', which selects lines that begin with a capital letter and end with a digit.
grep '^ERROR' app.log # lines starting with ERROR
grep '\.conf$' files.txt # lines ending in .conf
grep '^[0-9]' data.txt # lines starting with a digit
grep 'colou\{0,1\}r' text.txt # color or colour
Notice the backslashes in the last line. In the basic dialect, \{, \}, \(, \) and \| are the special forms, while the bare characters are literal. You will meet the cleaner extended dialect shortly.
Play with the pattern below:
grep -m 5 pattern prints five matches and stops reading, which is useful on a slow or embedded board scanning a large file. For a literal string with no regex meaning, grep -F is faster and matches the text exactly.
Going deeperHow grep matches: the engine
GNU grep compiles a basic or extended regex into a lazy deterministic finite automaton and simulates it in linear time, reporting the leftmost-longest match, as POSIX requires. That is why a plain grep never blows up the way a backtracking engine can on a bad pattern. For fixed strings, grep -F skips the automaton and uses an Aho-Corasick machine, or Boyer-Moore for a single literal, which is far faster. The backtracking approach, used by PCRE (grep -P) and many programming languages, tries one path and unwinds on failure, which on crafted patterns can take exponential time. This distinction is the subject of Russ Cox's paper in the references.
learner@embeddex:~/lab$ # grep matches the pattern anywhere, so noerror matches too learner@embeddex:~/lab$ grep error app.log an error occurred noerror here learner@embeddex:~/lab$ # -w requires a whole word, so noerror is left out learner@embeddex:~/lab$ grep -w error app.log an error occurred learner@embeddex:~/lab$ # -i ignores case, so ERROR matches too learner@embeddex:~/lab$ grep -i error app.log ERROR: disk full an error occurred noerror here learner@embeddex:~/lab$ # -c counts matching lines instead of printing them learner@embeddex:~/lab$ grep -c error app.log 2 learner@embeddex:~/lab$ # the ^ anchor matches only lines that start with ERROR learner@embeddex:~/lab$ grep '^ERROR' app.log ERROR: disk full
Inside the pipe
4 min readA pipe is not a file and not a queue that grows forever. The kernel allocates a circular buffer in memory and hands the process two descriptors, one for each end. On Linux the default capacity is 16 pages, or 64 KiB, and the constant PIPE_BUF sets the size, 4096 bytes, under which a write is guaranteed atomic.
the largest write to a pipe that is guaranteed atomic, 4096 bytes on Linux
The capacity is what makes pipelines safe on small machines. A fast producer filling the buffer blocks in write until the reader drains some bytes; a reader on an empty buffer blocks in read until the writer produces. The two commands run concurrently, and the pipe enforces backpressure, so a pipeline never buffers an entire file in memory.
the signal the kernel sends a process that writes to a pipe whose read end has been closed
The other edge case is a vanished reader. If the reading process exits and the writer keeps writing, the kernel delivers SIGPIPE, whose default action terminates the writer. The classic demonstration is yes, which writes forever:
yes | head -n 1
echo "${PIPESTATUS[0]}" # 141 in bash
head prints one line and exits, closing the read end. yes's next write hits a pipe with no reader, the kernel raises SIGPIPE, and yes dies with status 141, which is 128 plus the signal number 13.
A process can choose to ignore SIGPIPE and keep writing. Then write returns an error with errno set to EPIPE, which is why some programs print write error: Broken pipe instead of dying. Command-line tools mostly leave the default and simply stop, which is the behaviour you want when the reader has had enough and the rest of the output is useless.
The fixed size is a trade-off. A small buffer means a writer blocks sooner and the reader wakes sooner, giving low latency for interactive use; a large buffer lets a producer run ahead and amortises the cost of waking the consumer, giving higher throughput. Linux picked 64 KiB as a middle ground, and the size is tunable per pipe for workloads that need more.
Going deeperAtomic writes and resizing the buffer
Writes of PIPE_BUF bytes or fewer are atomic: interleaved writers cannot tear each other's data, so concurrent producers writing short records stay intact. Writes larger than PIPE_BUF may interleave with others. The capacity can be changed per pipe with fcntl(fd, F_SETPIPE_SZ, n), bounded by /proc/sys/fs/pipe-max-size; privileged processes may raise it beyond the default 64 KiB for high-throughput pipelines. The details live in the pipe(7) man page.
Buffering: why stdout lags stderr
3 min readTwo layers sit between your program and the bytes on disk. The kernel moves data through file descriptors directly, but the C standard library adds its own buffer on top, and that buffer is the source of surprising ordering.
holding output in a userspace buffer and writing it out in a batch rather than byte by byte
The library follows three rules. stdout is line buffered when it points at a terminal, meaning it flushes on every newline. stdout is fully buffered when it points at a pipe or a file, meaning it flushes only when its buffer fills or when the program exits. stderr is never buffered, so its bytes reach the device immediately.
The consequence shows up whenever you pipe a program that prints to both streams. stdout's output sits in the buffer while stderr's races straight through, so the error can appear before the result it belongs to:
python3 -c "import sys; print('out'); print('err', file=sys.stderr)" 2>&1 | cat
Typical output (yours will differ) shows err before out, because the piped stdout was fully buffered while stderr flushed at once. On the terminal, with stdout line buffered, they appear in the written order. The same program, the same lines, a different order.
You can change the buffering without editing the program. stdbuf -oL forces line buffering, stdbuf -o0 forces no buffering, and python3 -u does the same for Python.
stdbuf -oL cmd 2>&1 | cat
Run the earlier Python one-liner again with python3 -u (or set PYTHONUNBUFFERED=1) and the two lines come out in the written order even through the pipe. Note that stdbuf would not help here: it only changes the C library's stdio buffering, so it works for programs such as grep, cut and awk, while Python manages its own output buffers. The program did not change; only its buffering policy did, which is the point of the fix.
Going deeperWhy stderr is unbuffered by design
The C library sets stderr unbuffered so a diagnostic is seen even when the program crashes before flushing, since a buffered error could be lost with the process. The setvbuf call lets a program choose line, full or no buffering per stream, and stdbuf achieves the same from outside by preloading a library that overrides those choices. The reason the terminal gets line buffering while a pipe gets full buffering is an old performance trade-off: interactive output should feel immediate, while batch output benefits from fewer, larger writes.
Extended regular expressions
2 min readthe regex dialect of grep -E and egrep, in which +, ?, |, ( and { are operators without needing a backslash
The basic dialect forces a backslash before \|, \+, \?, \( and \{. The extended dialect, enabled with grep -E (or the old name egrep), makes those characters operators directly, which reads far better for real patterns.
grep -E 'error|warn|fatal' app.log # alternation
grep -E 'https?://' urls.txt # s is optional
grep -E 'colou?r' text.txt # color or colour
grep -E '[0-9]{4}-[0-9]{2}-[0-9]{2}' log # a date
grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' log # an IPv4 shape
Each operator deserves a line. | means or, matching if either side fits. ? makes the previous item optional, + means one or more, and * means zero or more. {n,m} repeats the previous item between n and m times. Parentheses group items so an operator can apply to the whole group, as in the repeated octet pattern above. Anchors ^ and $ and classes like [0-9] work the same in both dialects.
One thing that trips people switching from other languages: POSIX extended regular expressions have no \d. The portable spelling for a digit is [0-9] or the POSIX class [[:digit:]], alongside [[:alpha:]], [[:alnum:]] and [[:space:]].
Try the extended operators below:
Both dialects match greedily: they take the longest string the pattern allows, from the leftmost position where a match can start. There is no lazy or non-greedy operator in POSIX regular expressions, so there is no way to ask for the shortest match; that idea belongs to Perl-style engines, which GNU grep offers only through -P. Anchoring both ends turns a loose search into an exact check, so grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}$' accepts nothing but a bare IPv4 address with no surrounding text.
On an embedded root filesystem, grep is usually the BusyBox applet. It supports -E and -F but not -P, and its extended dialect is a subset of GNU's. Test a pattern on the target board, not just on your workstation, because an option that works in GNU grep may be absent on the device.
The text toolkit and the full pipeline
4 min readcut extracts columns. -d sets the delimiter and -f selects fields, while -c selects character positions.
cut -d: -f1 /etc/passwd # every username
cut -c1-10 access.log # the first ten characters
sort orders lines, with -n for numeric order, -r to reverse and -k to pick a key. uniq collapses adjacent duplicates, and -c prefixes each with a count. Because uniq only compares neighbours, you sort first. sort -u removes duplicates outright during sorting, a shortcut when you only want distinct values; reach for uniq -c when the count itself is the point.
sort words.txt | uniq -c # count each distinct word
sort -rn sizes.txt # largest first
tr translates, deletes or squeezes characters. It maps characters positionally, so tr 'A-Z' 'a-z' lowercases.
echo 'HELLO' | tr 'A-Z' 'a-z' # hello
tr -d '\r' < windows.txt # strip carriage returns
tr -s ' ' < messy.txt # collapse runs of spaces
xargs turns lines of stdin into arguments for another command. It is how a pipeline feeds a command that reads arguments, not stdin, and it splits the list into batches that fit within the kernel's command-line length limit.
grep -l 'TODO' *.c | xargs grep -n 'TODO'
find . -name '*.log' -print0 | xargs -0 grep -l error
-n bounds how many arguments each invocation receives, and -I{} substitutes a name at a fixed position, which matters when the argument must appear twice:
seq 1 20 | xargs -n 5 echo # five arguments per echo
echo report | xargs -I{} cp {} {}.bak
The kernel caps the size of a single command line, reported by getconf ARG_MAX. xargs respects it by splitting the input into batches, which is why grep TODO *.c over tens of thousands of files fails with argument list too long. Note that the glob is expanded by the shell before any command runs, so you cannot rescue it by piping its result into xargs; generate the names without one giant command line instead, with find . -name '*.c' -print0 | xargs -0 grep TODO or the bash builtin printf '%s ' *.c | xargs -0 grep TODO (a builtin is not subject to the limit).
The -print0 and -0 pair uses a NUL byte as the separator, which survives filenames containing spaces. Now the pieces combine into the log-analysis pipeline.
1extract one field
cut -d' ' -f1 access.log takes the first space-separated field of each line, the client IP.
2group the duplicates
sort puts identical IPs next to each other, so uniq -c can count them in one pass.
3rank and trim
sort -rn orders by count descending, and head keeps the top few.
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head
grep -c ' 404 ' access.log
Watch each stage transform the sample below:
The spaces around 404 stop the pattern matching 1404 or 4040. A real access log has spaces inside the quoted request line, which is why cut on a later field produces empty columns; field extraction that ignores whitespace runs is the job of awk, which the next lesson covers.
A board with 512 MB of RAM cannot slurp a multi-gigabyte log. The pipeline above streams line by line: only sort spills to disk when the input outgrows memory. Use sort -S to bound its memory or sort -T to point its temporary files at larger storage.
The same mistakes recur, and each has a symptom, a cause and a fix.
| Symptom | Cause | Fix |
|---|---|---|
Error still prints after cmd > f |
> redirects stdout only |
cmd > f 2>&1 or &> f |
f is empty after cmd 2>&1 > f |
duplicate ran before stdout moved | cmd > f 2>&1 |
> f wiped a file you meant to extend |
> truncates |
use >> |
uniq -c splits one value into several |
duplicates were not adjacent | sort | uniq -c |
grep word matches noerror too |
grep matches patterns, not words | grep -w word |
grep 'a|b' matches nothing |
| is alternation only in -E |
grep -E 'a|b' |
cut -d' ' -f2 gives empty fields |
every space is a delimiter | use awk, or tr -s ' ' first |
192.0.2.99
198.51.100.23
203.0.113.50
203.0.113.7
203.0.113.7
203.0.113.7
learner@embeddex:~/lab$ # uniq -c counts each run of adjacent duplicates
learner@embeddex:~/lab$ cut -d' ' -f1 access.log | sort | uniq -c
1 192.0.2.44
1 192.0.2.99
1 198.51.100.23
1 203.0.113.50
3 203.0.113.7
learner@embeddex:~/lab$ # sort -rn ranks by count, head keeps the top
learner@embeddex:~/lab$ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head
3 203.0.113.7
1 203.0.113.50
1 198.51.100.23
1 192.0.2.99
1 192.0.2.44
learner@embeddex:~/lab$ # the spaces around 404 stop it matching 1404 or 4040
learner@embeddex:~/lab$ grep -c ' 404 ' access.log
1Try it
1 min readOpen the lab for this session and run it in a scratch directory. Generate a small access log, then build the reports from this lesson: the top client IPs with cut -d' ' -f1 | sort | uniq -c | sort -rn | head, the 404 count with grep -c ' 404 ', and a redirection exercise that captures stdout and stderr separately with > f 2>&1. Then run the same pipeline over SSH on your Raspberry Pi and compare the BusyBox tools' options with the GNU versions. The practice pack adds predict-the-output drills for redirection order and buffering.
Key takeaways
- Every process gets three streams: stdin (0), stdout (1) and stderr (2).
- > truncates and >> appends; 2>&1 and &> capture stderr together with stdout.
- Redirections are applied left to right, so 2>&1 >file and >file 2>&1 mean different things.
- A pipe connects one process's stdout to the next's stdin; tee splits the stream to a file and onward.
- grep matches regular expressions anywhere in a line, not whole words; -w and -E change that.
- sort before uniq -c, because uniq only collapses adjacent identical lines.
- A pipe is a fixed-size kernel buffer that blocks the writer when full and delivers SIGPIPE when the reader is gone.
- stdout is line-buffered on a terminal but fully buffered in a pipe; stderr is never buffered.
Reflect
- Explain to a colleague, in two sentences: redirect stdout, stderr and stdin independently with >, >>, <, 2> and &>, and predict where each stream lands.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Official docsBash manual: Redirections
- Official docsGNU grep manual
- Official docsGNU Coreutils manual (cut, sort, uniq, tr)
- Man pagepipe(2) - create pipe
- Man pagepipe(7) - PIPE_BUF and capacity
- Man pagedup(2) - duplicate a file descriptor
- Man pagesetvbuf(3) - stream buffering
- SpecificationPOSIX.1-2017 regular expressions
- PaperRegular Expression Matching Can Be Simple And Fast
- Official docsBusyBox - The Swiss Army Knife of Embedded Linux
Official documentation first. Links checked when the lesson was published.
