Back to Embedded Linux
EmbeddexAi · Linux & Embedded Systems Engineering · Module 2

S09 — Pipes, FIFOs and redirection

pipe() · dup2 · named pipes · popen · a shell with | < > >> 2>
TL;DR — pipe() creates two fds (read end, write end) carrying a one-way byte stream. Close the ends you don't use, or read() never sees EOF. dup2() redirects stdin/stdout onto a pipe or file — the mechanism behind every shell pipeline and > redirection. A FIFO is a pipe with a name, so unrelated processes can meet at a path. popen() runs a command and returns its output as a stream.

1 · Overview

A pipe connects two processes with a one-way byte stream. It is the workhorse of the Unix philosophy: small tools, chained together. ls | wc -l works because the shell creates a pipe, points ls's stdout at its write end and wc's stdin at its read end. This session builds that mechanism from the primitives up, ending in a shell (mysh v3) that handles |, <, >, >>, and 2>.

2 · pipe() & the EOF rule

pipe(fd) returns two fds: fd[0] (read) and fd[1] (write). The rule that trips everyone: close the ends you do not use. read() returns EOF only when every write-end copy is closed — if a process (or a forked child) still holds an open write end, the reader blocks forever waiting for data that never comes.

$ ./pipe_demo pipe() -> read fd 3, write fd 4 [parent] got 86 bytes: fifo_demo ... pipe_demo.c popen_demo.c
/* child: point stdout at the pipe's write end, run ls */
dup2(fd[1], 1);          /* stdout = pipe write end */
close(fd[0]);            /* child does not read — close it */
close(fd[1]);
execlp("ls", "ls", NULL);

/* parent: read */
close(fd[1]);            /* CRITICAL: close the write end, or read() blocks */
read(fd[0], buf, sizeof buf);

3 · dup2 redirection

dup2(oldfd, newfd) makes newfd refer to whatever oldfd refers to, atomically (closing newfd first). Redirecting is therefore "dup2 onto fd 0/1/2":

dup2(fd[1], 1);   /* stdout  -> pipe/file */
dup2(infile, 0);  /* stdin   -> file      */
dup2(errf,   2);  /* stderr  -> file (2>) */

This is the whole of shell redirection: > opens a file and dup2s it onto 1; < onto 0; 2> onto 2.

4 · FIFOs

A FIFO (named pipe) is a pipe with a filesystem name, created with mkfifo. Unrelated processes meet by opening the same path — one for read, one for write. It must live on a filesystem that supports FIFOs:

$ mkfifo /tmp/m2s09-fifo && ls -l /tmp/m2s09-fifo prw-r--r-- 1 eslam eslam 0 ... /tmp/m2s09-fifo # 'p' = FIFO $ ./fifo_demo write /tmp/m2s09-fifo & $ ./fifo_demo read /tmp/m2s09-fifo writer wrote a line to /tmp/m2s09-fifo reader got: hello via FIFO
The /mnt/c trap WSL's /mnt/c is NTFS via drvfs, which does not support real FIFOs — mkfifo there does not create a working pipe and the writer blocks in open(). Run FIFOs on a real Linux filesystem (/tmp, ext4). This is a genuine, captured filesystem lesson.

5 · popen

popen() is the convenient wrapper: it forks a shell to run a command and returns a FILE* you can read (or write) — a pipe underneath.

$ ./popen_demo popen("ls -l | wc -l") -> 9 pclose status = 0
FILE *p = popen("ls -l | wc -l", "r");
fgets(buf, sizeof buf, p);
int status = pclose(p);   /* wait for the child, return its status */

6 · Code

6.1 mysh v3 — pipelines + redirection

/* split the line into segments on '|' */
char *segs[MAXSEG]; int nseg = 0;
char *s = strtok_r(line, "|", &save);
while (s) { segs[nseg++] = s; s = strtok_r(NULL, "|", &save); }

/* wire pipes between segments, fork each, dup2 in/out */
for (int i = 0; i < nseg; i++) {
    int in  = (i == 0) ? -1 : pipes[i-1][0];
    int out = (i == nseg-1) ? -1 : pipes[i][1];
    if (fork() == 0) { dup2(in, 0); dup2(out, 1); run_segment(segs[i]); }
}
$ printf "ls | wc -l\necho hi > out.txt\nls missing 2> err.txt\nexit\n" | ./mysh_v3 mysh> 9 mysh> hi mysh> ls: cannot access 'missing': No such file or directory

7 · Reference

int pipe(int fd[2])

Create a pipe; fd[0] read, fd[1] write.

int dup2(oldfd, newfd)

Make newfd a copy of oldfd (closing newfd first).

int mkfifo(path, mode)

Create a named pipe at path.

FILE *popen(cmd, mode) · int pclose(FILE *)

Run a shell command behind a pipe; read/write the stream; pclose waits.

8 · Pitfalls (8)

  1. Not closing unused pipe ends. The reader blocks forever — read() never sees EOF.
  2. Forgetting to close the write end in the parent. Even after the child exits, the parent's own copy keeps the pipe open.
  3. Using a FIFO on /mnt/c (drvfs). NTFS does not make real FIFOs — use /tmp (ext4).
  4. Opening a FIFO write end with no reader. open() blocks until a reader arrives (by design).
  5. Reading popen output without pclose. The child becomes a zombie; pclose waits and reports status.
  6. Assuming a pipe is bidirectional. It is one-way; for two-way use two pipes (or a socketpair).
  7. Redirecting > but forgetting O_TRUNC/O_APPEND. > truncates, >> appends — get the flag right.
  8. Leaking the pipe fds into the exec'd program. Set FD_CLOEXEC or close them before exec.

9 · Real-world (SensorNode)

The SensorNode gateway chains its tools with pipes: sampler | averager | uploader, each stage a small program wired by pipe()+dup2(). The config generator runs popen("date +%Y%m%d") to stamp a filename. And its CLI parser is mysh v3 — | to filter logs, 2> /var/log/sensornode/err to capture errors separately, all on the ext4 root (never on a drvfs mount, where FIFOs break).

10 · Check your understanding

Q1 — Why must both processes close the pipe ends they don't use?
read() returns EOF only when every write-end copy is closed. If any process holds an open write end, the reader blocks waiting for data that will never come.
Q2 — How does dup2 implement cmd > file?
Open file with O_WRONLY|O_CREAT|O_TRUNC, then dup2(fd, 1) — stdout now refers to the file. The command writes to fd 1 without knowing it is a file.
Q3 — What is a FIFO, and when do you use one instead of pipe()?
A FIFO is a named pipe — a pipe with a filesystem path. Use it when the two processes are unrelated (no parent/child link); pipe() only works between a parent and its forked child.
Q4 — Why does a FIFO fail on /mnt/c but work on /tmp?
/mnt/c is NTFS via drvfs, which does not implement FIFO semantics. /tmp is ext4, a real Linux filesystem that does. FIFOs need a filesystem that supports them.