Module 22 · Lesson 3 of 16

Files, subprocess, paths, logging: when Python replaces Bash

Replace fragile Bash with Python: pathlib, safe file I/O, subprocess without deadlocks or shell injection, and level-aware logging with rotation.

  • Beginner to advanced
  • 27 min read
  • 45 min lab
What you will need

Ubuntu 24.04 (WSL2, a VM, or a container) with Python 3.11 or newer (Ubuntu 24.04 ships 3.12); a terminal; no board needed for this lesson

By the end you can 7 objectives
  • Build and manipulate paths with pathlib: / joining, glob, stat, and the read and write helpers.
  • Read and write text and binary files safely with context managers, explicit encodings, and chunked reads.
  • Create safe temporary files and directories with tempfile, and replace a file atomically with os.replace.
  • Run external tools with subprocess.run, capture output, and handle return codes, timeouts, and exceptions.
  • Stream a child process with Popen and explain why naive pipe handling deadlocks.
  • Configure level-aware logging with handlers, formatters, and rotation, and decide between log and print.
  • Decide when Python should replace a Bash script and when a shell one-liner is still the right tool.
Read the lesson below; the video will appear here when published.
01

When Bash stops being enough

3 min read

In earlier modules you built pipelines like this one, which lists the ten largest logs:

bash
find /home/learner/logs -name '*.log' -printf '%s %p\n' | sort -rn | head -10

That line is good. It is short, it uses tools that already exist, and it does one job. Keep writing lines like this. The trouble starts when the job grows. Suppose the factory script must now retry a failed flash three times, give up after a timeout, parse the tool's JSON report, and write a summary that a test dashboard can read. In Bash, every one of those steps fights the language. Data is only text, so you cut it with awk and sed. Errors are only exit codes, so you check $? after every command or trust set -e, which has well-known corner cases. And the script must run on the build host, on a colleague's laptop, and inside a CI container, where the available shell may differ.

Python answers each of those pains directly. Data lives in real structures: lists, dicts, sets, and the json module. Errors raise exceptions you can catch, retry, or turn into a clean exit code. The same script runs unchanged anywhere Python runs. That is the trade this lesson teaches: Bash for quick glue, Python for tools with logic.

The standard library pieces you will use have a history worth knowing. Pipes themselves date to early-1970s UNIX, and Bash arrived in 1989 as the GNU replacement for the Bourne shell. Early Python could only run commands through os.system and os.popen, which were thin and awkward. PEP 324 added the subprocess module in Python 2.4 (2004), and it is still the right interface today. PEP 282 added the logging module in Python 2.3 (2003), modelled on log4j's ideas of levels and handlers. PEP 428 added pathlib in Python 3.4 (2014), replacing string surgery on paths with objects. Each of these replaced a generation of fragile glue code, and each is what this lesson covers.

The embedded angle is concrete. A provisioning script that flashes SensorNode boards on a factory line is a real program: it must handle a board that does not answer, a serial port that disappears, and a report that quality engineers read months later. That is Python work. Meanwhile the board itself may run only BusyBox ash, a small shell without Bash's arrays and [[ ]] tests, so a Bash script written on your laptop may not even run there. A Python tool on the host sidesteps the problem: you cross-compile firmware on the host, and the host's shell differences stop mattering.

Here is the rule of thumb you should leave with. Reach for Bash when the job is a short pipeline of existing tools, run once, by you. Reach for Python when the job has branches, loops over structured data, retries, timeouts, or output another program must parse. The rest of this lesson builds the four skills that make the switch: paths, files, subprocesses, and logging.

Bash versus PythonBashPythonDatatext, cutwith awklists, dicts,JSONErrorsexit codes,$?exceptionsyou catchPortabilityshell differsper hostsame codeeverywhereBest atquick pipes,one-linerslogic,retries,reports
Bash and Python compared on data, error handling, portability, and the jobs each is best at.
02

pathlib: paths as objects

4 min read

Start with the object itself. Path represents a filesystem path, and the / operator joins pieces onto it:

python
from pathlib import Path

logs = Path("/home/learner/logs")
boot = logs / "2026" / "boot.log"
print(boot)          # /home/learner/logs/2026/boot.log
print(boot.name)     # boot.log
print(boot.stem)     # boot
print(boot.suffix)   # .log
print(boot.parent)   # /home/learner/logs/2026

Notice what you did not do: no os.path.join, no manual slashes, no string concatenation. The result of / is a new Path, so you can keep chaining. Two class methods give you starting points: Path.home() is the current user's home directory, and Path.cwd() is the current working directory. Path() with no argument is the same as Path.cwd().

A path object also knows how to ask questions and do small jobs:

python
p = Path("/home/learner/logs/boot.log")
p.exists()           # True if anything is there
p.is_file()          # True only for a regular file
p.is_dir()           # True only for a directory
p.with_suffix(".txt")  # /home/learner/logs/boot.txt, a new Path
p.stat().st_size     # size in bytes
p.stat().st_mtime    # last modification time, seconds since the epoch

with_suffix is the clean way to derive an output name from an input name: boot.with_suffix(".json") gives the report path next to the log. stat() returns the file's metadata, the same information the stat command shows.

Finding files is where pathlib replaces find. iterdir() yields the immediate children of a directory. glob(pattern) matches children against a shell-style pattern, and rglob(pattern) does the same recursively through all subdirectories:

python
for path in logs.rglob("*.log"):
    size = path.stat().st_size
    kind = "large" if size > 1_000_000 else "small"
    print(f"{kind:5} {size:>10} {path}")

That loop is the heart of the lab: it replaces a find plus awk one-liner with code you can extend. Add a branch for symlinks, a date filter, or a JSON report, and you are writing a tool, not a line. Note that rglob does not descend into symlinked directories by default, which keeps it from wandering out of the tree you pointed it at. A is followed by most operations, but resolve() shows you the real target: it returns an absolute path with every symlink expanded. Since Python 3.6 it tolerates paths that do not exist yet, which is what you want when computing an output path before creating it.

Build and resolve the paths from this section below:

From /home/learner the relative path logs/2026/boot.log resolves to /home/learner/logs/2026/boot.log, the same result as the pathlib join logs / 2026 / boot.log. Try logs/current/boot.log instead: logs/current is a symlink to /var/log, so the path lands on /var/log/boot.log, which is the target resolve() reports.

  • cwd: /home/learner
  • value: logs/2026/boot.log

For quick content I/O, pathlib has shortcuts that open, read or write, and close in one call:

python
text = boot.read_text(encoding="utf-8")
boot.with_suffix(".txt").write_text(text.upper(), encoding="utf-8")
raw = boot.read_bytes()          # bytes, no decoding
Path("out.bin").write_bytes(raw)
Path("reports/2026").mkdir(parents=True, exist_ok=True)

Use these for small files: a config, a report, a device tree overlay. For large files or fine control over buffering, use open() directly, which is the next section. And when you maintain older code that passes paths as strings, os.path still works and accepts Path objects in most functions, so you can migrate one call at a time instead of all at once.

Anatomy of a Pathanchor/parent/home/learner/logsnameboot.logstembootsuffix.logparts = ('/', 'home', 'learner','logs', 'boot.log')
The fields of a Path object for /home/learner/logs/boot.log: anchor, parts, parent, name, stem, and suffix.
03

Opening, reading, and writing files

4 min read

Here is the shape you should use every time:

python
with open("/home/learner/logs/boot.log", "r", encoding="utf-8") as f:
    for line in f:
        if "error" in line:
            print(line.rstrip())

The first argument is the path, a string or a Path. The second is the mode: "r" read, "w" write (creates the file, truncates it if it exists), "a" append, and "x" create-only, which fails if the file already exists. Add "b" for binary or "t" for text; text is the default. So "rb" reads bytes and "w" writes text.

The with statement works because a file object is a . When the block ends, normally or through an exception, Python calls the file's __exit__, which closes it. That matters for two reasons. An open file holds a file descriptor, a small integer the kernel uses to track your open files, and a process has a limited supply. And data you wrote may sit in a buffer until close flushes it; a program that crashes without closing can leave a half-written file. with removes both risks, so never write f = open(...) without a with unless you have a specific reason and a try/finally.

Text mode decodes bytes into strings, and you must say how. An of UTF-8 is what almost every log, config, and source file on a modern Linux system uses. If you omit encoding=, Python picks one from the locale, which differs between machines and is a classic source of bugs that appear only in CI. Pass encoding="utf-8" explicitly, every time.

Binary mode for anything that is not text

Firmware images, device tree blobs, and captured serial traces are bytes, not characters. Open them with "rb" or "wb" and skip encoding entirely. Decoding binary data as text raises UnicodeDecodeError or, worse, silently corrupts it.

Newlines have one more subtlety. In text mode, reading translates \r\n (Windows) and lone \r (old Mac) into \n; this is called universal newlines. Writing with the default newline=None translates \n back to the platform's line ending. When you use the csv module, pass newline="" to open() so the module controls line endings itself, as its documentation requires.

Now the reading tools. f.read() returns the whole file as one string, f.read(n) returns at most n characters, f.readline() returns one line, and f.readlines() returns a list of all lines. Iterating the file object, as in the first example, yields one line at a time and is the memory-friendly default. For a multi-gigabyte boot log, never call read() or readlines(); read in chunks instead:

python
with open("capture.bin", "rb") as f:
    while chunk := f.read(65536):
        process(chunk)

The walrus operator assigns and tests in one step: the loop ends when read returns an empty bytes object at end of file. Writing is f.write(text), which returns the number of characters written and does not add a newline for you.

For whole-file operations, shutil is the high-level toolbox. shutil.copy(src, dst) copies content and permission bits; shutil.copy2 also preserves timestamps. shutil.copytree(src, dst, dirs_exist_ok=True) copies a directory tree, merging into an existing destination when the flag is set. shutil.move(src, dst) moves a file or tree, falling back to copy-plus-delete across filesystems. And shutil.rmtree(path) deletes a tree recursively; treat it as dangerous, check the path twice, and know that it refuses to run on a symlink rather than following it.

The layers of a writeTextIOWrapperstr to bytes, encodingBufferedWritercollects writes, flushesFileIOraw read and writeKernel page cacheStoragefsync reaches herea write passes down each layerTextIOWrapperstr to bytes, encodingBufferedWritercollects writes, flushesFileIOraw read and writeKernel page cacheStoragefsync reaches herea write passes down each layer
The layers a write passes through: the text wrapper, the buffer, the raw file object, the kernel page cache, and storage.
04

Temporary files and the os module

4 min read

Never invent a name like /tmp/mytool.tmp. Another process can predict it and create a symlink there first, pointing at a file you care about, so your write lands somewhere else. This is a real attack class, not a theoretical one. tempfile creates names that are random and opens them with flags that fail if the name already exists, so there is no gap to race:

python
import tempfile

with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as tmp:
    tmp.write("scratch data\n")
    scratch_path = tmp.name

with tempfile.TemporaryDirectory() as workdir:
    report = Path(workdir) / "report.json"
    report.write_text("{}", encoding="utf-8")
# the directory and everything in it are gone here

NamedTemporaryFile gives you a real file with a name. By default it is deleted when closed; delete=False keeps it so another tool can open it by name, and then cleanup is your job. On POSIX you can reopen the file by name while it is still open; on Windows you cannot, a platform difference worth remembering if your tool must run on both. TemporaryDirectory is the cleaner pattern when you need several scratch files: everything inside is removed when the with block ends, even on an exception. The files are created with owner-only permissions, so other users cannot read your scratch data.

The os module is the lower-level layer underneath. You will use a handful of its functions constantly:

python
import os

os.getcwd()                     # current working directory, a string
os.chdir("/home/learner/logs")  # change it; relative paths now resolve here
os.listdir(".")                 # entry names, not full paths, not recursive
os.makedirs("reports/2026", exist_ok=True)  # mkdir -p, no error if present
for dirpath, dirnames, filenames in os.walk("."):
    for name in filenames:
        full = os.path.join(dirpath, name)

os.walk yields one tuple per directory, top down, and you can prune the walk by editing dirnames in place. os.environ is a dict-like view of the environment, and changes to it are inherited by children you spawn later; the subprocess section builds on that. One warning about chdir: it changes the meaning of every relative path in the process, so a tool that changes directory and then opens a relative path it computed earlier is a common bug. Prefer absolute paths built with pathlib, and pass cwd= to subprocess instead of changing your own directory.

Older code passes paths as strings and uses os.path.join, os.path.exists, and friends. That is the bridge: os.path functions accept Path objects too, and os.fspath(p) converts a Path to a string when a library demands one. You do not need to rewrite legacy code to use pathlib in new code.

One pattern combines everything so far, and it is the pattern embedded engineers should know best: the atomic update. An matters when power can fail mid-write. A config file half-written by a power cut can brick a boot. The fix is to write a temporary file in the same directory, flush it to storage, then rename it over the target:

python
import os, tempfile

def atomic_write(path: Path, text: str) -> None:
    fd, tmp = tempfile.mkstemp(dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            f.write(text)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except BaseException:
        os.unlink(tmp)
        raise

os.replace is a rename that overwrites the destination, and on POSIX a rename within one filesystem is atomic: a reader sees either the old file or the new one, never a mix. The fsync before it pushes the data to storage, so the new content survives a power cut too. The rename itself being atomic does not mean the data is durable; that is what the fsync is for. Keep the temporary file in the same directory, because a rename across filesystems is a copy and loses the guarantee.

Check what /tmp is on the board

On many embedded systems /tmp is a tmpfs, a filesystem that lives in RAM. A large temporary file there consumes memory the device needs, and a tmpfs can be small. On a build host /tmp is usually disk. Check with findmnt /tmp, and pass dir= to tempfile when the scratch data must land on real storage.

The atomic update patternmkstemptemp insamedirectoryWrite andflushfsyncdatareachesstorageos.replaceatomicrename overtargetExit from step 2: any exceptionunlink temp, re-raisemkstemptemp insamedirectoryWrite andflushExit from step 2: anyexceptionunlink temp, re-raisefsyncdatareachesstorageos.replaceatomicrename overtarget
The atomic update pattern: write a temporary file, flush and fsync it, then os.replace over the target.
05

subprocess.run: running external tools

4 min read

Here is a complete, correct call:

python
import subprocess

result = subprocess.run(
    ["ls", "-l", "/home/learner/logs"],
    capture_output=True,
    text=True,
    check=True,
    timeout=10,
)
print(result.stdout)

Each keyword does one job. The first argument is a list: the program name, then each argument as its own element. capture_output=True collects the child's stdout and stderr instead of letting them flow to your terminal; it does this with s, which the next section opens up. text=True decodes the captured bytes to strings using the locale encoding; pass encoding="utf-8" to pin it. check=True raises an exception if the child exits non-zero. timeout=10 gives the child ten seconds.

The return value is a CompletedProcess with four fields you will use: args, returncode, stdout, and stderr. The return code follows the convention from the previous lesson: 0 means success, non-zero means failure, and distinct codes should mean distinct failures. Without check=True, a failed command is silent: run happily returns a result with returncode=1, and your script continues as if nothing happened. That is the single most common subprocess bug.

Wrap the call in try/except and handle each failure on its own terms:

python
try:
    result = subprocess.run(["fw_printenv", "-n", "bootcount"],
                            capture_output=True, text=True,
                            check=True, timeout=5)
except subprocess.CalledProcessError as e:
    print(f"tool failed with code {e.returncode}: {e.stderr}")
except subprocess.TimeoutExpired:
    print("tool did not answer in 5 seconds")
except FileNotFoundError:
    print("tool is not installed or not on PATH")

CalledProcessError carries the return code and the captured output, so you can log exactly what went wrong. TimeoutExpired means the deadline passed; run kills the child and waits for it before raising, so you do not leak a process. FileNotFoundError means the program itself could not be found, which is different from the program running and failing.

Now the rule that keeps your tool safe: never build a command by pasting strings into shell=True. With shell=True, your string is handed to /bin/sh -c, and the shell interprets it. If any part of the string comes from a filename, a user, or a board's response, you have a hole:

python
# dangerous: a file named 'x; rm -rf /home/learner/projects' runs that command
name = input("log file: ")
subprocess.run(f"grep error {name}", shell=True)

# safe: the name is one argument, whatever it contains
subprocess.run(["grep", "error", name])

With a list, the filename x; rm -rf ... is just a strange filename that grep cannot find. The shell never sees it. This is why the list form is the default and shell=True is the exception.

shell=True is a decision, not a convenience

Use shell=True only when you genuinely need shell syntax such as globs or pipes, and only with input you fully control. If a value must be embedded in a shell string, wrap it with shlex.quote first. Better: do the glob or the pipe in Python and keep the list form.

When you receive a command as a string, from a config file or a test fixture, shlex.split parses it into a list the way a shell would, respecting quotes:

python
import shlex
args = shlex.split("grep -i 'link up' boot.log")
# ['grep', '-i', 'link up', 'boot.log']
subprocess.run(args, capture_output=True, text=True)

And shlex.quote(value) does the reverse job: it wraps a value so a shell treats it as one literal word. Use it only when you cannot avoid shell=True.

What subprocess.run doesParentKernelChildclone + execchild startswrites outputpipes deliver bytesexit(status)returncode + output1. Parent → Kernelclone + exec2. Kernel → Childchild starts3. Child → Kernelwrites output4. Kernel → Parentpipes deliver bytes5. Child → Kernelexit(status)6. Kernel → Parentreturncode + output
What subprocess.run does: the parent asks the kernel to start the child, reads its output through pipes, and collects the exit status.
06

Popen: streaming and long-running children

5 min read

Popen starts the child and returns immediately, handing you an object with live streams:

python
proc = subprocess.Popen(
    ["tail", "-f", "/home/learner/logs/boot.log"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
)
for line in proc.stdout:
    print(line.rstrip())
    if "boot complete" in line:
        proc.terminate()
        break
proc.wait()

stdout=subprocess.PIPE creates a pipe whose read end becomes proc.stdout, a file object you can iterate line by line. poll() returns None while the child runs and its return code once it exits, so you can check without blocking. terminate() asks the child to stop (SIGTERM on POSIX), kill() forces it (SIGKILL), and after either you must call wait() or communicate() to collect the exit status; otherwise the child lingers as a zombie entry in the process table.

The dangerous call is wait() while output is piped and unread. To see why, you need one number: on Linux a pipe holds 65,536 bytes, 64 KiB, by default. When the child has written that much and you have not read any of it, the child's next write blocks. If your parent is blocked in wait(), it never reads. Neither side can move. That is a , and it is the classic subprocess failure: the script hangs, the child shows as sleeping, and nothing is logged.

Step by stepHow a piped child deadlocks its parent
  1. Parent starts child: Popen creates a pipe for stdout and launches the tool.

  2. Child writes output: the tool prints results into the pipe's write end.

  3. Pipe fills up: after 64 KiB unread, the kernel buffer is full.

  4. Child blocks: the next write cannot complete until space frees up.

  5. Parent blocks: wait() sleeps until the child exits, which it never will.

  6. Nothing moves: the parent must drain the pipe, but it is asleep. Use communicate() or read as data arrives.

communicate() is the safe one-shot answer. It reads stdout and stderr to the end, feeds stdin if you pass input=, and waits for the child, returning (stdout, stderr). It cannot deadlock because it drains the pipes while waiting. On POSIX it does this with a selector loop that watches both pipes at once; on Windows it uses threads, one per stream. If you need the output as one blob, use communicate() or just use run(), which calls it for you.

If you need the output line by line, as in the tail -f example, iterate the stream instead of waiting first. Two cautions. First, if you capture stderr separately and never read it, a chatty child can fill that pipe and deadlock the same way; merge it with stderr=subprocess.STDOUT when you do not need the streams apart, or drain both with selectors or threads. Second, the child may buffer its own output: many tools switch from line buffering to full buffering when stdout is a pipe, so lines arrive in bursts. For a C-stdio tool, stdbuf -oL forces line buffering; for a Python child, use python3 -u. stdbuf does not help programs that do their own buffering, such as Python without -u.

One more timeout distinction that bites people: run(timeout=...) kills the child for you, but communicate(timeout=...) and wait(timeout=...) only raise TimeoutExpired and leave the child running. After catching it, call proc.kill() and then proc.communicate() to reap it.

Going deeperPipe capacity, atomic writes, and SIGPIPE

The 64 KiB figure is a Linux fact with history. Before kernel 2.6.11 a pipe held one page, typically 4 KiB; since 2.6.11 it holds 16 pages, 65,536 bytes on systems with 4 KiB pages. Since 2.6.35 you can change a pipe's capacity with fcntl(fd, F_SETPIPE_SZ, size), up to /proc/sys/fs/pipe-max-size, which defaults to 1 MiB. None of this changes the deadlock rule: any finite buffer fills if nobody reads. Two related facts complete the picture. Writes of at most PIPE_BUF bytes (4096 on Linux) are atomic, so concurrent small writers do not interleave; larger writes may be split. And when every read end of a pipe is closed, a writer receives SIGPIPE, which kills it by default, or EPIPE if the signal is caught or ignored. That is why a pipeline like yes | head terminates cleanly: head exits, closes the read end, and yes dies on SIGPIPE instead of running forever.

The pipe deadlockParentChildPipe bufferPopen, stdout=PIPEwrites 64 KiBnext write blockswait() blocks too1. Parent → ChildPopen, stdout=PIPE2. Child → Pipe bufferwrites 64 KiB3. Child → Pipe buffernext write blocks4. Parent → Childwait() blocks too
The deadlock: the child blocks writing into a full pipe while the parent blocks in wait, and neither drains the buffer.
07

Composing pipelines and the environment

4 min read

Take grep error boot.log | wc -l. The in-process version needs no subprocess at all:

python
with open("boot.log", encoding="utf-8") as f:
    count = sum(1 for line in f if "error" in line)

That is fewer processes, no pipe to deadlock, and trivial error handling. Prefer it when the logic is simple. Call the real tools when they are faster or smarter than your Python: grep on a huge file, xz for compression, ssh for a remote host. To chain two tools, connect the first one's stdout to the second one's stdin:

python
p1 = subprocess.Popen(["grep", "error", "boot.log"],
                      stdout=subprocess.PIPE)
p2 = subprocess.Popen(["wc", "-l"],
                      stdin=p1.stdout, stdout=subprocess.PIPE, text=True)
p1.stdout.close()          # let p1 see SIGPIPE if p2 exits early
count, _ = p2.communicate()
p1.wait()

Closing p1.stdout in the parent matters: the parent holds a copy of the read end, and if it keeps it open, p1 never sees the pipe break when p2 exits. The child p2 still has its own copy, so the data flow is unaffected. This is the documented recipe for connecting two children, and it generalises to longer chains.

Run the same two stages on a log sample below:

The shell line grep error boot.log | wc -l is the cat, grep, wc chain here: of the ten sample lines grep keeps the three that contain the literal error, so wc reports 3. The match is a plain case-sensitive substring, exactly like grep without -i, and you can add or remove a stage to see what each tool contributes.

Every child inherits your environment unless you say otherwise. An is how tools find their configuration without command-line flags. os.environ is your process's environment, dict-like; set os.environ["LC_ALL"] = "C" and every child you spawn afterwards sees it. To give one child a different environment, pass env=:

python
env = os.environ.copy()
env["LC_ALL"] = "C"          # stable, English tool output for parsing
env["BOOT_COUNT"] = "3"
subprocess.run(["fw_printenv"], env=env, capture_output=True, text=True)

The critical word is copy. env= replaces the entire environment; it does not merge. Pass env={"BOOT_COUNT": "3"} and the child loses PATH, HOME, locale, everything else. Tools that look up helpers on PATH or read HOME then misbehave in ways that are hard to trace. Start from os.environ.copy() and change only what you mean to change. The same idea applies to cwd=, which sets the child's working directory without touching your own.

Command lookup follows the shell's rule. If the program name contains a slash, it is used as a path directly. If it has no slash, it is searched for in the directories of PATH, in order. shutil.which("fw_printenv") performs that same search and returns the full path, or None if the tool is absent, which is the clean way to check for a dependency before you need it. Be aware that some wrappers deliberately filter the environment: sudo, for example, strips variables such as LD_PRELOAD for security, so a child launched through it may not see what you set.

Going deeperHow CPython actually starts the child

Under subprocess.run on Linux, CPython normally forks and then execs the target program. It can take a posix_spawn fast path instead, but only when the options allow it: among other conditions, close_fds=False, no preexec_fn, no cwd, and an executable given with a directory component. With the defaults, you are on the fork-and-exec path. Two details follow. First, close_fds=True is the default, so the child inherits only the standard streams and any descriptors you pass explicitly; everything else is marked close-on-exec, which keeps your open sockets and serial ports out of the child. Second, if you watch with strace you will see clone or clone3, never a literal fork system call, because glibc implements fork with clone. The TimeoutExpired kill you saw earlier is SIGKILL sent to that child, followed by a wait to reap it.

A two-tool pipeline in PythonStart grepstdout=PIPEStart wc -lstdin=p1.stdoutCloseparent copyp1.stdout.close()communicate()drains wcoutputp1.wait()reap grepStart grepstdout=PIPEStart wc -lstdin=p1.stdoutCloseparent copyp1.stdout.close()communicate()drains wcoutputp1.wait()reap grep
A two-tool pipeline built with Popen: grep's stdout pipe becomes wc's stdin, and communicate drains the result.
08

logging: levels, handlers, and formatters

5 min read

Start with the difference. print writes text to stdout, with no severity, no timestamp, and no way to route it. A lets you keep detailed traces in a file while the console stays quiet, and change your mind later without editing code. The five standard levels, with their numeric values, are DEBUG (10), INFO (20), WARNING (30), ERROR (40), and CRITICAL (50). A message passes only if its level is at or above the threshold that is set.

The quickest setup is basicConfig, which configures the root logger:

python
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
log = logging.getLogger(__name__)
log.info("flashing %s", "fw.bin")
log.warning("boot count is %d", 3)

Two habits in that snippet matter. First, logging.getLogger(__name__) names the logger after the module, so in devkit/flash.py the logger is devkit.flash. Loggers form a hierarchy by dotted name, and a record propagates up to its ancestors' handlers, which is how one configuration covers a whole package. Second, pass values as arguments (log.info("flashing %s", name)) rather than building the string yourself; the formatting then happens only if the message will actually be emitted. One gotcha: basicConfig does nothing if the root logger already has handlers, which is why a second call in a test or a notebook appears to be ignored; pass force=True to reconfigure.

For a real tool you configure handlers explicitly. A is the destination, and a formatter shapes the text. There are two gates, and both must pass: the logger's level filters first, then each handler's level filters again. A logger set to DEBUG with a handler set to WARNING emits only WARNING and above through that handler. This two-gate design is what lets one logger feed a verbose file and a quiet console at the same time:

python
import logging
from logging.handlers import RotatingFileHandler

log = logging.getLogger("devkit")
log.setLevel(logging.DEBUG)

console = logging.StreamHandler()          # stderr by default
console.setLevel(logging.INFO)
console.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))

fileh = RotatingFileHandler("devkit.log", maxBytes=1_000_000,
                            backupCount=5, encoding="utf-8")
fileh.setLevel(logging.DEBUG)
fileh.setFormatter(logging.Formatter(
    "%(asctime)s %(levelname)s %(name)s: %(message)s"))

log.addHandler(console)
log.addHandler(fileh)

StreamHandler() writes to stderr, not stdout, which is the right default: your tool's real output stays on stdout for pipes, and diagnostics go to stderr. RotatingFileHandler solves the log that grows forever: when the file reaches maxBytes, it is renamed to devkit.log.1, older backups shift along, and a fresh file starts, keeping at most backupCount old files. TimedRotatingFileHandler rotates on a schedule instead, such as when="midnight". One caveat: rotation is designed for a single process; if several processes write one file, use a QueueHandler feeding one writer, or rotate externally.

For anything beyond a small script, put the configuration in data, not code. logging.config.dictConfig takes a dictionary, which you can load from JSON or build in one place:

python
import logging.config

logging.config.dictConfig({
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "plain": {"format": "%(levelname)s: %(message)s"},
        "full": {"format": "%(asctime)s %(levelname)s %(name)s: %(message)s"},
    },
    "handlers": {
        "console": {"class": "logging.StreamHandler",
                    "level": "INFO", "formatter": "plain"},
        "file": {"class": "logging.handlers.RotatingFileHandler",
                 "level": "DEBUG", "formatter": "full",
                 "filename": "devkit.log",
                 "maxBytes": 1000000, "backupCount": 5},
    },
    "loggers": {
        "devkit": {"level": "DEBUG", "handlers": ["console", "file"]},
    },
})

disable_existing_loggers=False keeps loggers that modules already created at import time; the default True silences them, a surprise worth knowing. The older fileConfig reads an ini-style file and still works, but dictConfig is the current recommendation. And if you log before configuring anything, Python uses a last-resort handler that writes WARNING and above to stderr, so early messages are not silently lost.

Step by stepThe journey of one log record
  1. Call: log.info creates a LogRecord with the message, level, time, and module.

  2. Logger gate: the record passes only if its level meets the logger's effective level.

  3. Handler gates: each attached handler checks the record against its own level.

  4. Format: the handler's formatter renders the record into text.

  5. Emit: the handler writes the text to its stream or file.

  6. Propagate: unless stopped, the record also travels to ancestor loggers' handlers.

Never log secrets

Logs outlive the run: they are copied, shipped to dashboards, and read by people who should not see credentials. Never log passwords, API tokens, private keys, or full connection strings with credentials. Log a fingerprint or a redacted form instead, and review what your DEBUG lines print before enabling them in production.

On a board, logging has a physical cost. Flash storage wears with every write, so a DEBUG-level file log on the device can shorten its life; keep verbose logs on the host, log to a tmpfs, or rate-limit. And if the root filesystem is read-only, point the file handler at a writable partition or drop the file handler entirely.

The logger hierarchyrootdevkitlevel DEBUGdevkit.flashrecord propagates upconsole handlerattached, INFO, stderrrotating fileattached, DEBUG, maxBytesrootdevkitlevel DEBUGdevkit.flashrecord propagates upconsole handlerattached, INFO, stderrrotating fileattached, DEBUG, maxBytes
The logger hierarchy: root, devkit, and devkit.flash, with console and rotating-file handlers attached to devkit.
09

mmap and the edge cases of paths

5 min read

A is what the mmap module exposes. Open the file in binary, then map it:

python
import mmap

with open("boot.log", "rb") as f:
    with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
        i = mm.find(b"kernel panic")
        if i != -1:
            print(mm[i:i + 200].decode("utf-8", errors="replace"))
        last = mm.rfind(b"error")

Length 0 means the whole file. The mapping behaves like a bytes object: find, rfind, slicing, and indexing all work, and mm[i] is an integer byte. Nothing is read up front. The kernel loads a page only when you first touch it, through a page fault, and evicts pages under memory pressure. That makes mmap excellent for random access and repeated scans of a huge file: you pay only for the regions you actually read, and the page cache is shared with every other process mapping the same file. For a single sequential pass, chunked reads are just as good and simpler.

Find which region an address falls in below:

A typical x86-64 layout for the reader process, with ASLR shifting these addresses on every run. The 4 MiB mapping of boot.log occupies 0x7f3c80000000 through 0x7f3c803fffff, in the wide gap between the heap below and the stack above. Entering 0x7f3c80001000 lands in that mapping at offset 4096, the second 4 KiB page of the file, which is exactly the page the kernel faults in when the program first touches it.

  • regions: [{"name": "Text", "start": "0x555555555000", "size": "0x1000", "detail": "Machine code of the reader binary, mapped read-only and executable"}, {"name": "Heap", "start": "0x555555559000", "size": "0x21000", "detail": "Small allocations; grows upward as brk is extended"}, {"name": "boot.log mapping", "start": "0x7f3c80000000", "size": "0x400000", "detail": "4 MiB of boot.log mapped read-only by mmap with length 0; a page is read only on first touch"}, {"name": "Stack", "start": "0x7ffffffde000", "size": "0x21000", "detail": "Grows downward from near the top of user space; one frame per active call"}]

The lifetime rules are where people get burned. On POSIX, the mapping stays valid even after you close the file descriptor; the mapping holds its own reference to the file. But the mmap object itself must stay alive: after mm.close(), any access raises ValueError. An empty file cannot be mapped at all and raises ValueError immediately, so check the size first. ACCESS_READ is read-only; ACCESS_WRITE requires the file opened "r+b" and writes go through to the file; ACCESS_COPY gives you private copy-on-write changes that never reach the disk. On Windows the rules differ, notably that an empty file cannot be mapped and the file cannot be resized while mapped, so test on every platform you ship to.

Going deepermmap hazards: truncation and durability

Two failure modes deserve a place in your mental model. First, if another process truncates the file while you have it mapped, touching a page beyond the new end raises SIGBUS, which kills the process by default; a log rotator that truncates in place can do this to your reader, which is one more reason to prefer rename-based rotation. Second, writes through an ACCESS_WRITE mapping land in the page cache, not necessarily on storage; call mm.flush() to push them, and remember that a crash before the kernel writes back can lose them, exactly as with buffered file writes. The mapping is a view of the page cache, not a durability guarantee.

Paths have their own edges, and most come from the difference between path math and filesystem reality. PurePath and its subclasses PurePosixPath and PureWindowsPath do path computation with no filesystem access at all, which is what you want when a Linux build host computes paths for a Windows machine or vice versa. Path itself is the concrete, platform-appropriate class: PosixPath on Linux, WindowsPath on Windows. Case is the classic trap: Linux filesystems are case-sensitive, so Boot.log and boot.log are different files; Windows and the default macOS filesystem are case-insensitive, so they are the same file. PureWindowsPath even compares case-insensitively to match Windows semantics, while PurePosixPath compares exactly. A tool that globs *.LOG on Linux and finds nothing has met this trap. Symlinks are the second: resolve() expands them, is_symlink() detects them, and rglob refuses to descend through them by default, so a tree walk stays inside the tree you named. When you need the truth about a path, ask the filesystem with stat() rather than trusting the string.

A file mapped into memoryhighStackgrows downmmap regionboot.log pages(free)Heapgrows upData / BSSlowTexthighStackgrows downmmap regionboot.log pages(free)Heapgrows upData / BSSlowText
A process address space with boot.log mapped into it: pages fault in on demand between the heap and the stack.
10

Common mistakes and how to fix them

2 min read

The script hangs with no output. Cause: a piped child filled the 64 KiB pipe buffer while the parent called wait() or communicate() was never reached. Diagnose: ps shows the child sleeping, and strace -p on the parent shows it blocked in wait4 while the child is blocked in write. Fix: drain the pipes with communicate(), or iterate the stream as data arrives; never wait() first with unread pipes.

Mojibake or UnicodeDecodeError on a file that looks fine. Cause: text mode decoded with the wrong encoding, often the locale default on a CI machine. Diagnose: open the file in binary and look at the offending bytes; check what encoding produced them. Fix: pass encoding="utf-8" explicitly, or the encoding the file actually uses; use binary mode for data that is not text.

The command works in your terminal but raises FileNotFoundError in the script. Cause: the script runs with a different environment or working directory, often because env= replaced the environment and dropped PATH, or a cron job starts with a minimal PATH. Diagnose: print shutil.which(tool) and os.environ["PATH"] inside the script. Fix: start env from os.environ.copy(), use absolute program paths, or check with shutil.which first.

The tool failed but the script carried on. Cause: no check=True and no look at returncode. Diagnose: log result.returncode and result.stderr after every call. Fix: check=True plus except subprocess.CalledProcessError, and branch on the code when different failures need different responses.

A filename turned into a command. Cause: shell=True with an f-string that interpolated untrusted text. Diagnose: search the codebase for shell=True and trace where every interpolated value comes from. Fix: pass a list; if a shell is unavoidable, shlex.quote every value.

Log lines are missing. Cause: the logger's level or the handler's level filters them out, or basicConfig ran after handlers already existed and did nothing. Diagnose: print log.getEffectiveLevel() and each handler's level. Fix: set both gates deliberately, and use force=True when reconfiguring.

Scratch files pile up in /tmp. Cause: NamedTemporaryFile(delete=False) without cleanup, or a crash before the cleanup line. Diagnose: list the prefix your tool uses. Fix: prefer TemporaryDirectory, whose cleanup runs even on exceptions, and keep scratch on the right filesystem with dir=.

Recurring failures and fixesScript hangspipe full, wait()firstcommunicate() orstreamUnicodeDecodeErrorwrong encodingencoding= or binaryFileNotFoundErrorPATH lost via env=os.environ.copy()Failureignoredno check=Truecheck=True + exceptFilename rancodeshell=Trueinjectionlist args,shlex.quoteLog linesmissinglevel gates filterset both gatesScratchpiles updelete=False, nocleanupTemporaryDirectorysymptom, cause, fix
The recurring failures of this lesson mapped from symptom to cause to fix.
11

Try it

1 min read

The lab has three parts that build on each other. First, write a pathlib script that walks a directory tree with rglob, classifies each file with stat, and prints a report, replacing a find plus awk one-liner. Second, run an external tool with subprocess.run, capture stdout and stderr, branch on the return code, and handle a timeout cleanly. Third, add level-aware logging with a console handler and a rotating file handler to that same script, then watch what each level gate lets through. The practice pack drills the same skills with shorter questions, including the deadlock and encoding traps. When you finish, the next lesson takes these tools to the hardware: serial ports with pyserial, SSH automation, and sockets.

Key takeaways

  • pathlib turns paths into objects: / joins, glob finds, stat inspects, and read_text or write_text does the I/O in one call.
  • with open(...) closes the file even when the block raises; always pass encoding= for text files.
  • tempfile creates files nobody can predict or race; write-temp-then-os.replace makes an update atomic.
  • Pass subprocess arguments as a list; shell=True hands your string to a shell and invites injection.
  • A Linux pipe holds 64 KiB by default; wait() before draining stdout can deadlock, so use communicate().
  • env= replaces the whole environment instead of merging, so start from os.environ.copy().
  • logging gives you levels, timestamps, and several destinations at once; print gives you none of them.
  • Bash still wins for quick pipes and one-liners; Python wins when logic, errors, and structured data appear.

Reflect

  1. Explain to a colleague, in two sentences: build and manipulate paths with pathlib: / joining, glob, stat, and the read and write helpers.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close