Connect programs with a stream of bytes
2 min readStart with a familiar command. This sends a directory listing to a program that counts newline characters.
ls | wc -l
The count depends on your directory. Filenames can contain newlines, so this is not a reliable general filename count.
A A pipe is a kernel-managed, one-way communication channel with separate read and write endpoints. The shell connects the first program's standard output to the second program's standard input. Neither program needs a special pipe-specific input format.
Standard input, standard output, and standard error normally use descriptors 0, 1, and 2. A descriptor is a small integer indexing a process's open-file table.
An ordinary pipe is a A byte stream is an ordered sequence of bytes with no built-in message boundaries. Suppose a producer writes AB, then CD. A consumer might receive ABCD together, or read A and later BCD. Neither write becomes a permanent record.
An ARM temperature producer and logger may run at different speeds. Separate readings with newlines or a length field stating the following record's byte count.
Create an anonymous pipe with this C interface, declared in <unistd.h>:
int pipe(int fd[2]);
Success returns zero and fills fd[0] with the read endpoint and fd[1] with the write endpoint. Failure returns minus one and sets errno. Treat the array entries as valid only after success. Do not assume their values are always 3 and 4.
Call read(fd[0], buffer, count) to consume bytes and write(fd[1], buffer, count) to produce them. Linux pipes do not support seeking; lseek() fails with ESPIPE. Their data lives in kernel memory, not a temporary disk file.
fork() shares anonymous-pipe descriptor references with the child. Named pipes let unrelated processes connect by pathname. Two-way conversations need two pipes or a Unix-domain socket pair.
BusyBox may supply both filters, but the kernel still implements their pipe connection.
Predict waiting, EOF, and broken pipes
4 min readImagine a parent sending a configuration string to its child. After pipe() and fork(), both processes hold references to both endpoints. The parent should close its read endpoint. The child should close its write endpoint.
The parent writes the configuration, then closes its write endpoint. The child reads the buffered data. A later nonzero-length read returns zero once the buffer is empty and no write endpoint remains open.
End of file, or EOF, means that a read returns zero because no more bytes are currently available from a completed input stream. For an anonymous pipe, completion requires all writer references to close. A zero-length read request is different: returning zero then proves nothing about completion.
Blocking calls wait for progress. Nonblocking calls return immediately when they cannot progress. This table assumes nonzero-length reads.
| Buffered bytes | Writer references | Blocking read | Nonblocking read |
|---|---|---|---|
| Some | Any | Returns available bytes, up to request | Same |
| None | At least one | Waits | Fails with EAGAIN |
| None | None | Returns zero | Returns zero |
Reading fewer bytes than requested is normal. Process that count, then read again. If a signal interrupts the call before any transfer, handle EINTR according to your shutdown policy. Never parse bytes that the call did not actually return.
Closing the original write descriptor is insufficient if dup() created another reference. A child or grandchild can also keep the stream alive after the original producer exits. EOF depends on references, not a particular process's lifetime.
In the opposite direction, writing when no read endpoints remain generates SIGPIPE. Its default action terminates the writer. If SIGPIPE is ignored, the failed write reports EPIPE. Blocking the signal makes it pending; that is different from ignoring it. A large write that already transferred data needs careful handling of its positive partial count too.
The signal lesson established child signal policy. Keep that policy when adding pipelines: commands should normally receive default SIGPIPE behavior. Otherwise yes | head -n 1 can produce an unexpected broken-pipe diagnostic rather than ordinary producer termination.
Create: The parent creates one read endpoint and one write endpoint.
Fork: The child inherits references to both endpoints.
Separate: The parent closes its read copy; the child closes its write copy.
Transfer: The parent writes while the child reads and processes bytes.
Finish: The parent closes its writer; the child drains remaining bytes and then sees EOF.
Reap: The parent collects the child's termination status after communication can finish.
Redirect descriptors in the right order
3 min readThese commands create, append, read, and capture errors. Run them in a disposable directory.
printf 'first\n' > readings.txt
printf 'second\n' >> readings.txt
wc -l < readings.txt
ls /path-that-does-not-exist 2> errors.txt
The first replaces contents; the second appends. The third reads through descriptor 0. The fourth redirects descriptor 2 alone.
| Operator | Target descriptor | Required open flags |
|---|---|---|
< path |
0 | O_RDONLY |
> path |
1 | O_WRONLY, O_CREAT, O_TRUNC |
>> path |
1 | O_WRONLY, O_CREAT, O_APPEND |
2> path |
2 | O_WRONLY, O_CREAT, O_TRUNC |
For creation, pass a mode such as 0666; the process's permission mask, umask, removes permissions. O_APPEND places each write at the current end of a regular file. It does not make a multi-call log record indivisible or force data to stable storage.
dup(oldfd) returns the lowest unused descriptor referring to the same open file description. That description holds shared state, including status flags. dup2(oldfd, newfd) instead chooses the destination number and atomically replaces an existing mapping there.
Do not substitute close(newfd); dup(oldfd);: another thread or a handler could allocate the freed number. With valid equal arguments, dup2(fd, fd) changes nothing. See dup(2).
1Open, duplicate, and release the extra handle
This helper belongs in a single-threaded mysh child before execution. It checks duplication and handles the case where open() already returned the target descriptor.
static int install_fd(int source, int target)
{
if (source == target)
return 0;
int result;
do {
result = dup2(source, target);
} while (result == -1 && errno == EINTR);
return result == -1 ? -1 : 0;
}
static int redirect_file(const char *path, int target, int flags)
{
int fd = open(path, flags, 0666);
if (fd == -1)
return -1;
int result = install_fd(fd, target);
int saved = errno;
if (fd != target)
(void)close(fd);
errno = saved;
return result;
}
Include <errno.h>, <fcntl.h>, and <unistd.h>. These opens omit O_CLOEXEC: when fd == target, it would remove the standard descriptor during exec. Alternatively, explicitly clear FD_CLOEXEC in that case.
2Preserve the command's left-to-right meaning
Bash command > combined.txt 2>&1 opens a file for descriptor 1, then duplicates that mapping onto descriptor 2. Reversing the redirections leaves descriptor 2 pointing to the original output destination. 2>&1 copies a mapping at that moment; it does not link two descriptor numbers forever.
Bash &> combined.txt redirects both output streams. Use > combined.txt 2>&1 for portable POSIX-shell syntax. Our required mysh v3 subset implements the five operators <, >, >>, 2>, and |; it should reject unsupported operators clearly.
A here-document supplies inline input. This Bash example quotes the delimiter, so the body undergoes no variable or command expansion.
cat <<'END_INPUT'
sensor=$SENSOR
status=ready
END_INPUT
An unquoted delimiter permits shell expansions. <<- strips leading tab characters, not arbitrary spaces. A future mysh implementation must collect the body before launch. Filling a pipe synchronously before starting its reader can deadlock on a long document; use a temporary file or a concurrent writer. These forms are documented in the Bash redirection manual.
Launch two filters without waiting too early
3 min readFor ls | wc -l, create one pipe and two children. The first child maps the write endpoint onto descriptor 1. The second maps the read endpoint onto descriptor 0. Each closes both original pipe descriptors after duplication.
The parent closes both endpoints and supervises; only the children move data.
This complete teaching program requires descriptors 0, 1, and 2 to be open, default signal behavior, and no competing child reaper.
#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>
static void collect(pid_t pid)
{
while (waitpid(pid, NULL, 0) == -1) {
if (errno != EINTR) {
perror("waitpid");
exit(EXIT_FAILURE);
}
}
}
static void run_filter(int p[2], int source, int target,
const char *program, char *const args[])
{
if (dup2(source, target) == -1) {
perror("dup2");
_exit(126);
}
(void)close(p[0]);
(void)close(p[1]);
execvp(program, args);
int saved = errno;
perror(program);
_exit(saved == ENOENT ? 127 : 126);
}
int main(void)
{
int p[2];
if (pipe(p) == -1) {
perror("pipe");
return EXIT_FAILURE;
}
pid_t producer = fork();
if (producer == -1) {
perror("fork");
(void)close(p[0]);
(void)close(p[1]);
return EXIT_FAILURE;
}
if (producer == 0) {
char *args[] = {"ls", NULL};
run_filter(p, p[1], STDOUT_FILENO, "ls", args);
}
pid_t consumer = fork();
int fork_error = errno;
if (consumer == 0) {
char *args[] = {"wc", "-l", NULL};
run_filter(p, p[0], STDIN_FILENO, "wc", args);
}
(void)close(p[0]);
(void)close(p[1]);
if (consumer == -1) {
errno = fork_error;
perror("fork");
(void)kill(producer, SIGKILL);
collect(producer);
return EXIT_FAILURE;
}
collect(producer);
collect(consumer);
return EXIT_SUCCESS;
}
Save it as two_filters.c. Compile with warnings, then compare its count with the shell pipeline in the same unchanged directory.
cc -std=c11 -Wall -Wextra -Wpedantic two_filters.c -o two_filters
./two_filters
ls | wc -l
This program reports launch failures, without shell exit-status rules. A shell retains every stage's status. Bash normally uses the last stage; pipefail changes that choice.
A leaked read endpoint in the producer prevents SIGPIPE when the consumer exits. An extra writer complicates EOF detection.
Going deeperDescriptors can collide with standard input and output
If a launcher starts with descriptor 0 or 1 closed, a new pipe can reuse that number. Blindly closing both originals after dup2() could then close the installed standard stream. Establish valid standard descriptors before allocating pipes, or move internal descriptors above 2 with F_DUPFD_CLOEXEC and handle equal-number duplication explicitly. A service can attach missing standard streams to /dev/null; an interactive shell should verify its terminal setup instead.
Cross-compile with your board's SDK, the toolchain and libraries prepared for that target. Run the binary on Linux; a host executable cannot test ARM behavior.
Give unrelated processes a FIFO name
4 min readA A FIFO, short for first in, first out, is a named pipe opened through a filesystem pathname. mkfifo() creates the name, not a running reader or writer. Its C interfaces are:
#include <sys/stat.h>
int mkfifo(const char *path, mode_t mode);
int mkfifoat(int dirfd, const char *path, mode_t mode);
For mkfifoat(), include <fcntl.h> and define _POSIX_C_SOURCE 200809L before headers. A relative path is resolved against the directory descriptor dirfd. An absolute path ignores it. Both functions return zero on success or minus one on failure. Creation permissions are reduced by umask.
1Observe the open handshake
Create a private directory in terminal A. Keep the printed pathname so you can use it in terminal B. mktemp -d chooses a fresh directory; verify command success before continuing.
labdir=$(mktemp -d /tmp/fifo-lab.XXXXXX) || exit 1
chmod 700 "$labdir"
mkfifo -m 600 "$labdir/events" || exit 1
printf '%s\n' "$labdir"
cat "$labdir/events"
cat waits while opening for reading. In terminal B, replace the example directory with the actual printed path and send one record.
printf 'temperature=24\n' > /tmp/fifo-lab.REPLACE/events
The two opens meet. cat receives the line, then exits after that writer closes and the buffer drains. Afterwards, remove your FIFO and directory with rm "$labdir/events" and rmdir "$labdir" in terminal A.
| Open request | Peer absent | Peer present |
|---|---|---|
| O_RDONLY | Waits for writer | Succeeds |
| O_WRONLY | Waits for reader | Succeeds |
| O_RDONLY with O_NONBLOCK | Succeeds | Succeeds |
| O_WRONLY with O_NONBLOCK | Fails with ENXIO | Succeeds |
A nonblocking read from an empty FIFO returns zero if there are no writers. It returns EAGAIN if a writer remains. A future writer can reconnect to an existing FIFO, so EOF need not end a long-running server's lifetime. Define whether the server closes and reopens, or intentionally holds a dummy writer. A dummy writer suppresses EOF and therefore changes disconnect detection.
Linux allows opening a FIFO with O_RDWR without a peer. POSIX does not define this behavior. It can hide peer disappearance or let a process read its own bytes; do not mistake it for a portable connection handshake. See fifo(7).
2Separate naming, framing, and replies
For a server, use one well-known request FIFO and a private reply FIFO per client. The client creates and opens its reply reader nonblocking before submitting a request. The server opens that reply FIFO for writing with O_NONBLOCK, so a vanished client cannot stall every other client.
A protocol defines how peers exchange and interpret data. Framing is an application rule that identifies where each record starts and ends within a byte stream. Define a request length, identifier, and validated reply name. Encode bytes explicitly: C structure padding and byte order can differ across boards.
Keep each complete request within PIPE_BUF and send it with one write(). The server accumulates read bytes and extracts whole records; one read can split a request or combine several. Reject oversized lengths before allocating memory. One server reader is simplest: multiple readers compete for bytes rather than each receiving a copy.
Bound client retries, reply sizes, and outstanding requests. Opening a reply FIFO may fail with ENXIO because the client disappeared. A later write can still fail with EPIPE. These are ordinary lifecycle failures, not permission failures.
Place production FIFOs in a service-owned directory under /run, commonly backed by tmpfs. tmpfs is a memory-backed filesystem whose data can use RAM and, when available, swap. The FIFO node's placement controls naming and permissions; FIFO payloads still use kernel pipe buffers. /run contents disappear across reboot, so recreate the directory and node at startup.
FIFOs also work on supported disk filesystems; ext4 is not a requirement. Some shared or Windows-backed mounts reject FIFO creation. Inspect the actual mount and error rather than recommending root access. On WSL, use the Linux filesystem for this lab. In a shared /tmp, never trust an existing predictable name after EEXIST. A private directory prevents unrelated users replacing that name.
Distinguish atomicity, capacity, and kernel storage
4 min readA An atomic pipe write inserts its bytes without another writer's bytes appearing inside that write. It does not guarantee delivery to one read call, persistence, or a particular reader. Linux defines PIPE_BUF as 4096 bytes. Portable code can query fpathconf(fd, _PC_PIPE_BUF) and handle its return value rather than assuming Linux's constant.
The following table assumes readers remain, a positive request size, and no signal or other exceptional interruption.
| Write size | Blocking descriptor | O_NONBLOCK descriptor |
|---|---|---|
| At most PIPE_BUF | Waits until the whole request fits; inserts it atomically | Whole request succeeds, or EAGAIN with no bytes transferred |
| Greater than PIPE_BUF | Can wait and interleave with other writers | Can return a positive partial count; EAGAIN when no progress is possible |
For one shared request FIFO, split-header-then-body writes lose the atomic-record guarantee. Another client can insert its request between them. Build the entire bounded record first. If one atomic write gets EINTR before any bytes move, retry the same record. Do not resume an unexpectedly partial record as though a new atomic write could restore the original guarantee.
Capacity is bounded but adjustable. Define _GNU_SOURCE before headers for Linux-specific fcntl operations. Run this fragment after creating p successfully.
int current = fcntl(p[0], F_GETPIPE_SZ);
if (current == -1) {
perror("F_GETPIPE_SZ");
} else {
int granted = fcntl(p[0], F_SETPIPE_SZ, current);
if (granted == -1)
perror("F_SETPIPE_SZ");
}
Passing current makes the example conservative. For a real change, choose a measured burst budget. The returned size may be rounded up. Shrinking below buffered usage can fail with EBUSY. Permission and per-user pipe-memory limits can reject growth. Inspect /proc/sys/fs/pipe-max-size, pipe-user-pages-soft, and pipe-user-pages-hard where exposed.
Buffer-page arrangement can prevent using every nominally free byte. Large pipes consume board resources and increase queueing delay while hiding slow consumers.
To enable nonblocking behavior on an existing endpoint, preserve its other status flags:
int flags = fcntl(p[0], F_GETFL);
if (flags == -1 || fcntl(p[0], F_SETFL, flags | O_NONBLOCK) == -1)
perror("enable nonblocking read");
Duplicated descriptors share these status flags through their open file description. FD_CLOEXEC, in contrast, belongs to each descriptor. pipe2(p, O_CLOEXEC | O_NONBLOCK) sets flags when creating the pipe, avoiding a later inheritance race. Use O_CLOEXEC alone for shell data pipelines: ordinary filters often expect blocking input and output.
Going deeperFollow the actual buffer and wait queues
In Linux source, start with fs/pipe.c, include/linux/pipe_fs_i.h, and fs/splice.c. struct pipe_inode_info holds pipe state, reader and writer counts, buffer-ring positions, locking, and wait queues. A wait queue tracks tasks sleeping until an event can let them proceed. Each struct pipe_buffer describes a page-backed segment through a page reference, offset, length, flags, and operations. The kernel synchronizes updates and wakes waiting peers as data, space, or endpoint presence changes. These are implementation structures, not a stable user-space ABI, meaning a promised binary interface for applications. Read the source matching your board's kernel.
Packet-mode pipes are a Linux exception to ordinary byte-stream behavior. pipe2(p, O_DIRECT) preserves packets from writes, splitting writes larger than PIPE_BUF. A too-small read buffer discards the unread remainder of that packet. Zero-length packets are unsupported. This flag does not mean storage direct I/O, does not provide network packets, and should not be enabled for ordinary shell pipelines. See pipe2 flags.
Parse and execute mysh v3 pipelines
5 min readKeep v2's single child-status owner. Do not add waitpid() while a handler independently reaps the same children.
1Extend the lexer, then build a command plan
A lexer turns text into tokens, the individual words and operators a parser consumes. Preserve quote information until it decides whether characters are operators. printf 'a|b' contains one argument with a pipe character, not two stages. Match >> before > and recognize the adjacent descriptor prefix in 2>.
Add token kinds END, WORD, BAR, INPUT, OUTPUT, APPEND, and ERROR_OUTPUT. Each WORD owns a null-terminated string until launch completes. Have the lexer reject unsupported control operators, unfinished quotes, and unsupported descriptor numbers. An isolated argument 2 stays a word. In echo 2 > file, the space means 2 is an argument; echo 2>file redirects descriptor 2.
These definitions extend your lexer. Avoid strtok(), which loses adjacent-operator and quoting information.
enum kind { END, WORD, BAR, INPUT, OUTPUT, APPEND, ERROR_OUTPUT };
struct token { enum kind kind; char *text; };
enum { MAX_STAGES = 16, MAX_ARGS = 64, MAX_REDIRS = 32 };
struct redir { int target, flags; char *path; };
struct command {
char *argv[MAX_ARGS + 1];
size_t argc, nr;
struct redir redirs[MAX_REDIRS];
};
/* Tokens include END; words remain alive through launch. */
static int parse_pipeline(const struct token *t,
struct command c[MAX_STAGES], size_t *count)
{
memset(c, 0, sizeof(*c) * MAX_STAGES);
size_t stage = 0, k = 0;
for (;;) {
struct command *cmd = &c[stage];
enum kind op = t[k].kind;
if (op == END || op == BAR) {
if (cmd->argc == 0)
return -1;
if (op == END) {
*count = stage + 1;
return 0;
}
if (++stage == MAX_STAGES)
return -1;
++k;
} else if (op == WORD) {
if (cmd->argc == MAX_ARGS)
return -1;
cmd->argv[cmd->argc++] = t[k++].text;
} else {
if (cmd->nr == MAX_REDIRS || t[k + 1].kind != WORD)
return -1;
struct redir r = {.path = t[k + 1].text};
switch (op) {
case INPUT:
r.target = 0; r.flags = O_RDONLY; break;
case OUTPUT:
r.target = 1;
r.flags = O_WRONLY | O_CREAT | O_TRUNC; break;
case APPEND:
r.target = 1;
r.flags = O_WRONLY | O_CREAT | O_APPEND; break;
case ERROR_OUTPUT:
r.target = 2;
r.flags = O_WRONLY | O_CREAT | O_TRUNC; break;
default:
return -1;
}
cmd->redirs[cmd->nr++] = r;
k += 2;
}
}
}
Include <stddef.h>, <string.h>, and <fcntl.h>. Zero-initialization leaves each argument array terminated by NULL. The explicit limits make resource use visible; reject overflow rather than silently truncating commands. This teaching grammar rejects redirection-only commands such as > file, although Bash accepts them.
An ordered redirection list matters even without descriptor-duplication syntax. printf x > first > second must create or truncate first before routing output to second. Storing only the last output filename would lose a visible side effect.
2Connect a rolling pair of stages
N stages need N minus one pipes. A rolling launcher holds only the previous read endpoint and next pipe, reducing simultaneous descriptor use.
The following child-side helper uses install_fd() and redirect_file() from the redirection section. It assumes all internal descriptors exceed 2. Its caller has already restored the command's signal dispositions and mask from v2.
static void execute_stage(const struct command *cmd,
int previous, int next_read, int next_write)
{
if ((previous != -1 && install_fd(previous, 0) == -1) ||
(next_write != -1 && install_fd(next_write, 1) == -1)) {
perror("pipeline dup2");
_exit(126);
}
if (previous != -1) (void)close(previous);
if (next_read != -1) (void)close(next_read);
if (next_write != -1) (void)close(next_write);
for (size_t j = 0; j < cmd->nr; ++j) {
const struct redir *r = &cmd->redirs[j];
if (redirect_file(r->path, r->target, r->flags) == -1) {
perror(r->path);
_exit(126);
}
}
execvp(cmd->argv[0], cmd->argv);
int saved = errno;
perror(cmd->argv[0]);
_exit(saved == ENOENT ? 127 : 126);
}
Wire the helper into your existing launch loop in this order. pipe2() requires _GNU_SOURCE and <fcntl.h> on Linux. This is the launch algorithm; status ownership and failure cleanup remain in your v2 main loop.
Prepare: Parse every stage, validate limits, and block SIGCHLD while registering children.
Allocate: Start with previous equal to minus one; create the next pipe with O_CLOEXEC unless this is the last stage.
Fork: In the child, restore command signal policy and call execute_stage with the previous endpoint and next endpoints.
Release: In the parent, record the PID, close previous and the next write endpoint, then retain the next read endpoint.
Repeat: Launch the following stage immediately; never wait within the launch loop.
Collect: After the final stage, restore the parent's signal mask and let the one status owner collect every registered PID.
Pipeline wiring happens before explicit redirections. In printf x > capture | wc -c, the file replaces the producer's pipe output. The downstream reader sees EOF once all pipe writers close. In printf x | wc -c < input, the consumer instead reads input; the producer may get SIGPIPE.
On a pipe or fork failure halfway through launch, close every endpoint still held by the parent. Cancel and reap the already recorded children, then restore the saved mask. Keep failed-launch children registered until collected. A bounded teaching cleanup can send SIGKILL to those PIDs; never target unrelated processes. Full terminal process-group management belongs to the next lesson.
Builtins add another decision. A standalone cd must change the parent shell's directory. A builtin in a pipeline normally runs in a child and cannot change the parent. Parent-executed redirections require saving, replacing, then restoring standard descriptors on every path. Keep that extension separate from external-command pipeline wiring.
Choose popen or page-based transfer deliberately
3 min readpopen() provides a C library stream connected to a shell command. popen(command, "r") reads its standard output; "w" writes its standard input. It returns a FILE *, the buffered C stream handle, or NULL. It runs /bin/sh -c, whose syntax may differ from Bash.
This complete function reads fixed-command output and checks both stream errors and termination. Include _POSIX_C_SOURCE 200809L before headers, plus <stdio.h> and <sys/wait.h>.
static int count_with_shell(void)
{
FILE *stream = popen("printf 'board\\n' | wc -c", "r");
if (stream == NULL)
return -1;
char line[128];
while (fgets(line, sizeof line, stream) != NULL)
fputs(line, stdout);
int read_failed = ferror(stream);
int status = pclose(stream);
if (read_failed || status == -1)
return -1;
if (!WIFEXITED(status) || WEXITSTATUS(status) != 0)
return -1;
return 0;
}
pclose() closes the stream and waits for the associated process. Its result is an encoded wait status, not directly the exit code. fclose() cannot replace that wait. A generic reaper must not steal this child's status. With write mode, buffered data may remain unsent until fflush() or pclose(), and a reader that exits early can cause SIGPIPE.
Untrusted filenames concatenated into shell text can become executable syntax. Use explicit argument arrays with fork/exec for such inputs. Reserve popen for fixed, trusted commands.
For bulk forwarding, Zero-copy describes a path that avoids some data copying, commonly the transfer through an application-owned buffer. It does not promise no copying anywhere. Devices, filesystems, and fallback kernel paths determine actual costs.
splice() transfers between descriptors with at least one pipe endpoint. Pipe offset arguments must be NULL. A file-to-file forwarder therefore uses file to pipe, then pipe to file. Track the positive byte count, drain exactly that amount, and handle partial transfers, EINTR, and EAGAIN. Appending output files are not supported splice destinations. splice(2) defines these restrictions.
The system call tee() duplicates bytes between two pipes without consuming the input bytes. It differs from the shell utility named tee. After duplicating K bytes into a logging branch, consume exactly those K bytes from the original pipe before duplicating the next range. Otherwise you can log the same bytes repeatedly. A slow branch still needs a bounded queue or an explicit drop policy. See tee(2).
vmsplice() supplies user-memory ranges through an array of struct iovec, each containing an address and length. For a write endpoint, Linux can retain references to user pages. Treat that memory as unchanged until downstream use finishes. Reading a pipe into user memory does not provide the same zero-copy benefit. SPLICE_F_GIFT adds strict page alignment and ownership requirements; it is not a casual optimization. See vmsplice(2).
Going deeperMeasure the path rather than trusting the name
The kernel's splice documentation includes paths that copy into pages before linking them into a pipe. Page references can reduce user/kernel copies while still incurring allocation, locking, scheduling, and device transfers. On an ARM SoC, a system combining CPU and peripherals, cache effects and memory bandwidth may dominate. Pipe pages are not automatically buffers suitable for a hardware DMA engine, which transfers data without CPU byte-by-byte instructions. Measure throughput, CPU use, memory use, and end-to-end delay on the actual board before accepting the added lifetime complexity.
Diagnose stalls and validate the result
5 min readThe producer writes more than the available pipe space. It waits for a reader. The parent waits for that producer before starting the consumer. Nobody can perform the action another process needs. This is a A deadlock is a situation where participants cannot progress because each waits for an action the others cannot perform.
The same failure appears with two request/reply pipes. If both peers write messages larger than available capacity before reading, both can block. Size alone is not a protocol. Use a request-then-response order, concurrent draining, or readiness-driven nonblocking I/O.
A readiness event only says an operation may progress. It does not reserve enough room for your next request. Keep unsent bytes and retry after the next writable event. The later I/O-multiplexing lesson develops poll() and epoll; neither removes the need for these counters.
| Symptom | Likely cause | Diagnose | Fix |
|---|---|---|---|
| Consumer waits after producer exit | Leaked writer reference | Inspect every descendant's descriptors | Close inherited and duplicated writers |
| Producer hangs only with larger input | Parent waits before launching reader | Trace fork, write, and wait order | Launch all stages before waiting |
| Producer does not get SIGPIPE | Unused reader remains open | Inspect parent and producer endpoints | Close unused readers everywhere |
| FIFO startup waits | Peer has not opened opposite direction | Trace openat and inspect peer lifetime | Start peers concurrently or use defined nonblocking retry |
| FIFO records contain mixed data | Record exceeds PIPE_BUF or uses several writes | Trace write sizes and framing | Send each bounded record in one write |
| Pipeline output arrives in bursts | User-space buffering | Check producer flush policy | Flush deliberately or select a compatible buffering mode |
| Command succeeds but shell reports ECHILD | Competing status owners | Trace all waits and SIGCHLD paths | Centralize reaping or reserve ownership |
1Follow endpoint numbers through real system calls
Trace your shell and its children to find endpoint owners. This command prints diagnostics to the terminal.
strace -f -e trace=pipe,pipe2,dup2,dup3,close,openat,execve,read,write,wait4,waitid ./mysh
Look for pipe creation, each duplication, and each original descriptor's closure. Follow both forked processes and executed programs. libc may use different underlying process-creation calls, so add %process to the filter if investigating creation itself. Tracing affects timing; repeat without it after fixing a suspected race.
Replace the PID with your process's actual number. Descriptor links reveal pipe identifiers; fdinfo provides flags.
target=12345
ls -l "/proc/$target/fd"
cat "/proc/$target/fdinfo/1"
cat "/proc/$target/wchan"
Match pipe identifiers across the shell and descendants. A wait-channel name can suggest a blocked kernel operation, but kernel configuration and access restrictions can hide it. An identifier alone does not reveal endpoint direction; inspect descriptor flags and trace its origin.
For delayed producer output, remember that changing the consumer cannot flush the producer's C library buffer. GNU stdbuf applies to compatible C stdio programs; Python generally needs python3 -u. BusyBox images may lack stdbuf and strace. Add diagnostics to a development image or reproduce with the same target libraries and kernel configuration.
2Compare behavior with Bash on controlled input
Run equivalent commands in Bash and mysh in separate fresh directories. Compare file bytes, stderr destinations, and final status using deterministic input.
printf 'gamma\nalpha\nbeta\n' > input.txt
cat < input.txt | sort | head -n 2 > result.txt
printf 'tail\n' >> result.txt
wc -l < result.txt
ls /path-that-does-not-exist 2> errors.txt
Try the cat | sort | head -n 2 stages on the same input:
Then exercise quoted filenames, adjacent operators, empty stages, missing redirection targets, and inaccessible files. Invalid syntax must fail before spawning children. Redirection failure must prevent that child from executing its command. Successful parent execution must leave the shell's own prompt connected correctly.
Test more data than the measured capacity, an early-exiting consumer, and repeated pipelines under a small descriptor limit. Confirm that descriptor counts return to baseline and no zombies remain. A zombie is an exited child whose retained status has not yet been collected. Keep deliberate hanging tests in disposable processes you can terminate.
Try it
1 min readUse the lab, mysh v3: pipelines, ordered redirection, and FIFO peers, to implement ls | wc, then extend your v2 shell to bounded multi-stage pipelines and the five required operators. Compare supported commands against Bash and retain evidence of descriptor closure, correct file bytes, and complete child collection. In the practice pack, predict FIFO blocking and nonblocking opens, analyze leaked endpoints, and distinguish atomicity from capacity. Finish by choosing a framing rule for a board's FIFO service and explaining its limits before moving to Job control; mysh complete (mini project).
Key takeaways
- An ordinary pipe carries ordered bytes, without preserving write boundaries.
- An empty pipe reaches EOF only after every writer reference disappears.
- dup2 changes a descriptor mapping; it does not copy file contents.
- Launch every pipeline stage before waiting, and close unused endpoints everywhere.
- FIFO open behavior depends on direction, peer presence, and O_NONBLOCK.
- PIPE_BUF limits atomic writes; it does not describe total pipe capacity.
- Apply explicit redirections after pipeline wiring, in their original order.
- Zero-copy interfaces still need ownership, backpressure, and partial-transfer handling.
Reflect
- Explain to a colleague, in two sentences: identify pipe endpoints and explain byte-stream behavior.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Man pagepipe(7): byte streams, capacity, and atomicity
- Man pagepipe(2): pipe and pipe2 flags
- Man pagedup(2): descriptor duplication
- Man pagefifo(7): named-pipe open behavior
- Man pagemkfifo(3): mkfifo and mkfifoat
- Man pagepopen(3): command streams and pclose
- Official docsGNU Bash manual: redirections
- Official docsLinux kernel: splice and pipes
- Man pagesplice(2): transfer rules and limitations
Official documentation first. Links checked when the lesson was published.