When Bash stops being enough
3 min readIn earlier modules you built pipelines like this one, which lists the ten largest logs:
find /home/learner/logs -name '*.log' -printf '%s %p\n' | sort -rn | head -10
That line is good. It is short, it uses tools that already exist, and it does one job. Keep writing lines like this. The trouble starts when the job grows. Suppose the factory script must now retry a failed flash three times, give up after a timeout, parse the tool's JSON report, and write a summary that a test dashboard can read. In Bash, every one of those steps fights the language. Data is only text, so you cut it with awk and sed. Errors are only exit codes, so you check $? after every command or trust set -e, which has well-known corner cases. And the script must run on the build host, on a colleague's laptop, and inside a CI container, where the available shell may differ.
Python answers each of those pains directly. Data lives in real structures: lists, dicts, sets, and the json module. Errors raise exceptions you can catch, retry, or turn into a clean exit code. The same script runs unchanged anywhere Python runs. That is the trade this lesson teaches: Bash for quick glue, Python for tools with logic.
The standard library pieces you will use have a history worth knowing. Pipes themselves date to early-1970s UNIX, and Bash arrived in 1989 as the GNU replacement for the Bourne shell. Early Python could only run commands through os.system and os.popen, which were thin and awkward. PEP 324 added the subprocess module in Python 2.4 (2004), and it is still the right interface today. PEP 282 added the logging module in Python 2.3 (2003), modelled on log4j's ideas of levels and handlers. PEP 428 added pathlib in Python 3.4 (2014), replacing string surgery on paths with objects. Each of these replaced a generation of fragile glue code, and each is what this lesson covers.
The embedded angle is concrete. A provisioning script that flashes SensorNode boards on a factory line is a real program: it must handle a board that does not answer, a serial port that disappears, and a report that quality engineers read months later. That is Python work. Meanwhile the board itself may run only BusyBox ash, a small shell without Bash's arrays and [[ ]] tests, so a Bash script written on your laptop may not even run there. A Python tool on the host sidesteps the problem: you cross-compile firmware on the host, and the host's shell differences stop mattering.
Here is the rule of thumb you should leave with. Reach for Bash when the job is a short pipeline of existing tools, run once, by you. Reach for Python when the job has branches, loops over structured data, retries, timeouts, or output another program must parse. The rest of this lesson builds the four skills that make the switch: paths, files, subprocesses, and logging.
pathlib: paths as objects
4 min readStart with the object itself. Path represents a filesystem path, and the / operator joins pieces onto it:
from pathlib import Path
logs = Path("/home/learner/logs")
boot = logs / "2026" / "boot.log"
print(boot) # /home/learner/logs/2026/boot.log
print(boot.name) # boot.log
print(boot.stem) # boot
print(boot.suffix) # .log
print(boot.parent) # /home/learner/logs/2026
Notice what you did not do: no os.path.join, no manual slashes, no string concatenation. The result of / is a new Path, so you can keep chaining. Two class methods give you starting points: Path.home() is the current user's home directory, and Path.cwd() is the current working directory. Path() with no argument is the same as Path.cwd().
A path object also knows how to ask questions and do small jobs:
p = Path("/home/learner/logs/boot.log")
p.exists() # True if anything is there
p.is_file() # True only for a regular file
p.is_dir() # True only for a directory
p.with_suffix(".txt") # /home/learner/logs/boot.txt, a new Path
p.stat().st_size # size in bytes
p.stat().st_mtime # last modification time, seconds since the epoch
with_suffix is the clean way to derive an output name from an input name: boot.with_suffix(".json") gives the report path next to the log. stat() returns the file's metadata, the same information the stat command shows.
Finding files is where pathlib replaces find. iterdir() yields the immediate children of a directory. glob(pattern) matches children against a shell-style pattern, and rglob(pattern) does the same recursively through all subdirectories:
for path in logs.rglob("*.log"):
size = path.stat().st_size
kind = "large" if size > 1_000_000 else "small"
print(f"{kind:5} {size:>10} {path}")
That loop is the heart of the lab: it replaces a find plus awk one-liner with code you can extend. Add a branch for symlinks, a date filter, or a JSON report, and you are writing a tool, not a line. Note that rglob does not descend into symlinked directories by default, which keeps it from wandering out of the tree you pointed it at. A a directory entry that points to another path by name, rather than to the file's data directly is followed by most operations, but resolve() shows you the real target: it returns an absolute path with every symlink expanded. Since Python 3.6 it tolerates paths that do not exist yet, which is what you want when computing an output path before creating it.
Build and resolve the paths from this section below:
For quick content I/O, pathlib has shortcuts that open, read or write, and close in one call:
text = boot.read_text(encoding="utf-8")
boot.with_suffix(".txt").write_text(text.upper(), encoding="utf-8")
raw = boot.read_bytes() # bytes, no decoding
Path("out.bin").write_bytes(raw)
Path("reports/2026").mkdir(parents=True, exist_ok=True)
Use these for small files: a config, a report, a device tree overlay. For large files or fine control over buffering, use open() directly, which is the next section. And when you maintain older code that passes paths as strings, os.path still works and accepts Path objects in most functions, so you can migrate one call at a time instead of all at once.
Opening, reading, and writing files
4 min readHere is the shape you should use every time:
with open("/home/learner/logs/boot.log", "r", encoding="utf-8") as f:
for line in f:
if "error" in line:
print(line.rstrip())
The first argument is the path, a string or a Path. The second is the mode: "r" read, "w" write (creates the file, truncates it if it exists), "a" append, and "x" create-only, which fails if the file already exists. Add "b" for binary or "t" for text; text is the default. So "rb" reads bytes and "w" writes text.
The with statement works because a file object is a an object with enter and exit methods that a with statement runs on the way in and on the way out, even if the block raises. When the block ends, normally or through an exception, Python calls the file's __exit__, which closes it. That matters for two reasons. An open file holds a file descriptor, a small integer the kernel uses to track your open files, and a process has a limited supply. And data you wrote may sit in a buffer until close flushes it; a program that crashes without closing can leave a half-written file. with removes both risks, so never write f = open(...) without a with unless you have a specific reason and a try/finally.
Text mode decodes bytes into strings, and you must say how. An the rule that maps bytes to characters, such as UTF-8 or Latin-1 of UTF-8 is what almost every log, config, and source file on a modern Linux system uses. If you omit encoding=, Python picks one from the locale, which differs between machines and is a classic source of bugs that appear only in CI. Pass encoding="utf-8" explicitly, every time.
Firmware images, device tree blobs, and captured serial traces are bytes, not characters. Open them with "rb" or "wb" and skip encoding entirely. Decoding binary data as text raises UnicodeDecodeError or, worse, silently corrupts it.
Newlines have one more subtlety. In text mode, reading translates \r\n (Windows) and lone \r (old Mac) into \n; this is called universal newlines. Writing with the default newline=None translates \n back to the platform's line ending. When you use the csv module, pass newline="" to open() so the module controls line endings itself, as its documentation requires.
Now the reading tools. f.read() returns the whole file as one string, f.read(n) returns at most n characters, f.readline() returns one line, and f.readlines() returns a list of all lines. Iterating the file object, as in the first example, yields one line at a time and is the memory-friendly default. For a multi-gigabyte boot log, never call read() or readlines(); read in chunks instead:
with open("capture.bin", "rb") as f:
while chunk := f.read(65536):
process(chunk)
The walrus operator assigns and tests in one step: the loop ends when read returns an empty bytes object at end of file. Writing is f.write(text), which returns the number of characters written and does not add a newline for you.
For whole-file operations, shutil is the high-level toolbox. shutil.copy(src, dst) copies content and permission bits; shutil.copy2 also preserves timestamps. shutil.copytree(src, dst, dirs_exist_ok=True) copies a directory tree, merging into an existing destination when the flag is set. shutil.move(src, dst) moves a file or tree, falling back to copy-plus-delete across filesystems. And shutil.rmtree(path) deletes a tree recursively; treat it as dangerous, check the path twice, and know that it refuses to run on a symlink rather than following it.
Temporary files and the os module
4 min readNever invent a name like /tmp/mytool.tmp. Another process can predict it and create a symlink there first, pointing at a file you care about, so your write lands somewhere else. This is a real attack class, not a theoretical one. tempfile creates names that are random and opens them with flags that fail if the name already exists, so there is no gap to race:
import tempfile
with tempfile.NamedTemporaryFile(mode="w", suffix=".log", delete=False) as tmp:
tmp.write("scratch data\n")
scratch_path = tmp.name
with tempfile.TemporaryDirectory() as workdir:
report = Path(workdir) / "report.json"
report.write_text("{}", encoding="utf-8")
# the directory and everything in it are gone here
NamedTemporaryFile gives you a real file with a name. By default it is deleted when closed; delete=False keeps it so another tool can open it by name, and then cleanup is your job. On POSIX you can reopen the file by name while it is still open; on Windows you cannot, a platform difference worth remembering if your tool must run on both. TemporaryDirectory is the cleaner pattern when you need several scratch files: everything inside is removed when the with block ends, even on an exception. The files are created with owner-only permissions, so other users cannot read your scratch data.
The os module is the lower-level layer underneath. You will use a handful of its functions constantly:
import os
os.getcwd() # current working directory, a string
os.chdir("/home/learner/logs") # change it; relative paths now resolve here
os.listdir(".") # entry names, not full paths, not recursive
os.makedirs("reports/2026", exist_ok=True) # mkdir -p, no error if present
for dirpath, dirnames, filenames in os.walk("."):
for name in filenames:
full = os.path.join(dirpath, name)
os.walk yields one tuple per directory, top down, and you can prune the walk by editing dirnames in place. os.environ is a dict-like view of the environment, and changes to it are inherited by children you spawn later; the subprocess section builds on that. One warning about chdir: it changes the meaning of every relative path in the process, so a tool that changes directory and then opens a relative path it computed earlier is a common bug. Prefer absolute paths built with pathlib, and pass cwd= to subprocess instead of changing your own directory.
Older code passes paths as strings and uses os.path.join, os.path.exists, and friends. That is the bridge: os.path functions accept Path objects too, and os.fspath(p) converts a Path to a string when a library demands one. You do not need to rewrite legacy code to use pathlib in new code.
One pattern combines everything so far, and it is the pattern embedded engineers should know best: the atomic update. An an operation that other observers see as either fully done or not started, never half done matters when power can fail mid-write. A config file half-written by a power cut can brick a boot. The fix is to write a temporary file in the same directory, flush it to storage, then rename it over the target:
import os, tempfile
def atomic_write(path: Path, text: str) -> None:
fd, tmp = tempfile.mkstemp(dir=path.parent)
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(text)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
except BaseException:
os.unlink(tmp)
raise
os.replace is a rename that overwrites the destination, and on POSIX a rename within one filesystem is atomic: a reader sees either the old file or the new one, never a mix. The fsync before it pushes the data to storage, so the new content survives a power cut too. The rename itself being atomic does not mean the data is durable; that is what the fsync is for. Keep the temporary file in the same directory, because a rename across filesystems is a copy and loses the guarantee.
On many embedded systems /tmp is a tmpfs, a filesystem that lives in RAM. A large temporary file there consumes memory the device needs, and a tmpfs can be small. On a build host /tmp is usually disk. Check with findmnt /tmp, and pass dir= to tempfile when the scratch data must land on real storage.
subprocess.run: running external tools
4 min readHere is a complete, correct call:
import subprocess
result = subprocess.run(
["ls", "-l", "/home/learner/logs"],
capture_output=True,
text=True,
check=True,
timeout=10,
)
print(result.stdout)
Each keyword does one job. The first argument is a list: the program name, then each argument as its own element. capture_output=True collects the child's stdout and stderr instead of letting them flow to your terminal; it does this with a kernel buffer with a write end and a read end, used to pass a byte stream from one process to anothers, which the next section opens up. text=True decodes the captured bytes to strings using the locale encoding; pass encoding="utf-8" to pin it. check=True raises an exception if the child exits non-zero. timeout=10 gives the child ten seconds.
The return value is a CompletedProcess with four fields you will use: args, returncode, stdout, and stderr. The return code follows the convention from the previous lesson: 0 means success, non-zero means failure, and distinct codes should mean distinct failures. Without check=True, a failed command is silent: run happily returns a result with returncode=1, and your script continues as if nothing happened. That is the single most common subprocess bug.
Wrap the call in try/except and handle each failure on its own terms:
try:
result = subprocess.run(["fw_printenv", "-n", "bootcount"],
capture_output=True, text=True,
check=True, timeout=5)
except subprocess.CalledProcessError as e:
print(f"tool failed with code {e.returncode}: {e.stderr}")
except subprocess.TimeoutExpired:
print("tool did not answer in 5 seconds")
except FileNotFoundError:
print("tool is not installed or not on PATH")
CalledProcessError carries the return code and the captured output, so you can log exactly what went wrong. TimeoutExpired means the deadline passed; run kills the child and waits for it before raising, so you do not leak a process. FileNotFoundError means the program itself could not be found, which is different from the program running and failing.
Now the rule that keeps your tool safe: never build a command by pasting strings into shell=True. With shell=True, your string is handed to /bin/sh -c, and the shell interprets it. If any part of the string comes from a filename, a user, or a board's response, you have a a flaw where untrusted text is interpreted as shell syntax, letting it run commands you never intended hole:
# dangerous: a file named 'x; rm -rf /home/learner/projects' runs that command
name = input("log file: ")
subprocess.run(f"grep error {name}", shell=True)
# safe: the name is one argument, whatever it contains
subprocess.run(["grep", "error", name])
With a list, the filename x; rm -rf ... is just a strange filename that grep cannot find. The shell never sees it. This is why the list form is the default and shell=True is the exception.
Use shell=True only when you genuinely need shell syntax such as globs or pipes, and only with input you fully control. If a value must be embedded in a shell string, wrap it with shlex.quote first. Better: do the glob or the pipe in Python and keep the list form.
When you receive a command as a string, from a config file or a test fixture, shlex.split parses it into a list the way a shell would, respecting quotes:
import shlex
args = shlex.split("grep -i 'link up' boot.log")
# ['grep', '-i', 'link up', 'boot.log']
subprocess.run(args, capture_output=True, text=True)
And shlex.quote(value) does the reverse job: it wraps a value so a shell treats it as one literal word. Use it only when you cannot avoid shell=True.
Popen: streaming and long-running children
5 min readPopen starts the child and returns immediately, handing you an object with live streams:
proc = subprocess.Popen(
["tail", "-f", "/home/learner/logs/boot.log"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
for line in proc.stdout:
print(line.rstrip())
if "boot complete" in line:
proc.terminate()
break
proc.wait()
stdout=subprocess.PIPE creates a pipe whose read end becomes proc.stdout, a file object you can iterate line by line. poll() returns None while the child runs and its return code once it exits, so you can check without blocking. terminate() asks the child to stop (SIGTERM on POSIX), kill() forces it (SIGKILL), and after either you must call wait() or communicate() to collect the exit status; otherwise the child lingers as a zombie entry in the process table.
The dangerous call is wait() while output is piped and unread. To see why, you need one number: on Linux a pipe holds 65,536 bytes, 64 KiB, by default. When the child has written that much and you have not read any of it, the child's next write blocks. If your parent is blocked in wait(), it never reads. Neither side can move. That is a a state where two or more parties each wait for the other to act, so none of them ever proceeds, and it is the classic subprocess failure: the script hangs, the child shows as sleeping, and nothing is logged.
Parent starts child: Popen creates a pipe for stdout and launches the tool.
Child writes output: the tool prints results into the pipe's write end.
Pipe fills up: after 64 KiB unread, the kernel buffer is full.
Child blocks: the next write cannot complete until space frees up.
Parent blocks: wait() sleeps until the child exits, which it never will.
Nothing moves: the parent must drain the pipe, but it is asleep. Use communicate() or read as data arrives.
communicate() is the safe one-shot answer. It reads stdout and stderr to the end, feeds stdin if you pass input=, and waits for the child, returning (stdout, stderr). It cannot deadlock because it drains the pipes while waiting. On POSIX it does this with a selector loop that watches both pipes at once; on Windows it uses threads, one per stream. If you need the output as one blob, use communicate() or just use run(), which calls it for you.
If you need the output line by line, as in the tail -f example, iterate the stream instead of waiting first. Two cautions. First, if you capture stderr separately and never read it, a chatty child can fill that pipe and deadlock the same way; merge it with stderr=subprocess.STDOUT when you do not need the streams apart, or drain both with selectors or threads. Second, the child may buffer its own output: many tools switch from line buffering to full buffering when stdout is a pipe, so lines arrive in bursts. For a C-stdio tool, stdbuf -oL forces line buffering; for a Python child, use python3 -u. stdbuf does not help programs that do their own buffering, such as Python without -u.
One more timeout distinction that bites people: run(timeout=...) kills the child for you, but communicate(timeout=...) and wait(timeout=...) only raise TimeoutExpired and leave the child running. After catching it, call proc.kill() and then proc.communicate() to reap it.
Going deeperPipe capacity, atomic writes, and SIGPIPE
The 64 KiB figure is a Linux fact with history. Before kernel 2.6.11 a pipe held one page, typically 4 KiB; since 2.6.11 it holds 16 pages, 65,536 bytes on systems with 4 KiB pages. Since 2.6.35 you can change a pipe's capacity with fcntl(fd, F_SETPIPE_SZ, size), up to /proc/sys/fs/pipe-max-size, which defaults to 1 MiB. None of this changes the deadlock rule: any finite buffer fills if nobody reads. Two related facts complete the picture. Writes of at most PIPE_BUF bytes (4096 on Linux) are atomic, so concurrent small writers do not interleave; larger writes may be split. And when every read end of a pipe is closed, a writer receives SIGPIPE, which kills it by default, or EPIPE if the signal is caught or ignored. That is why a pipeline like yes | head terminates cleanly: head exits, closes the read end, and yes dies on SIGPIPE instead of running forever.
Composing pipelines and the environment
4 min readTake grep error boot.log | wc -l. The in-process version needs no subprocess at all:
with open("boot.log", encoding="utf-8") as f:
count = sum(1 for line in f if "error" in line)
That is fewer processes, no pipe to deadlock, and trivial error handling. Prefer it when the logic is simple. Call the real tools when they are faster or smarter than your Python: grep on a huge file, xz for compression, ssh for a remote host. To chain two tools, connect the first one's stdout to the second one's stdin:
p1 = subprocess.Popen(["grep", "error", "boot.log"],
stdout=subprocess.PIPE)
p2 = subprocess.Popen(["wc", "-l"],
stdin=p1.stdout, stdout=subprocess.PIPE, text=True)
p1.stdout.close() # let p1 see SIGPIPE if p2 exits early
count, _ = p2.communicate()
p1.wait()
Closing p1.stdout in the parent matters: the parent holds a copy of the read end, and if it keeps it open, p1 never sees the pipe break when p2 exits. The child p2 still has its own copy, so the data flow is unaffected. This is the documented recipe for connecting two children, and it generalises to longer chains.
Run the same two stages on a log sample below:
Every child inherits your environment unless you say otherwise. An a named string value, like PATH or HOME, that a process receives from its parent and can read at run time is how tools find their configuration without command-line flags. os.environ is your process's environment, dict-like; set os.environ["LC_ALL"] = "C" and every child you spawn afterwards sees it. To give one child a different environment, pass env=:
env = os.environ.copy()
env["LC_ALL"] = "C" # stable, English tool output for parsing
env["BOOT_COUNT"] = "3"
subprocess.run(["fw_printenv"], env=env, capture_output=True, text=True)
The critical word is copy. env= replaces the entire environment; it does not merge. Pass env={"BOOT_COUNT": "3"} and the child loses PATH, HOME, locale, everything else. Tools that look up helpers on PATH or read HOME then misbehave in ways that are hard to trace. Start from os.environ.copy() and change only what you mean to change. The same idea applies to cwd=, which sets the child's working directory without touching your own.
Command lookup follows the shell's rule. If the program name contains a slash, it is used as a path directly. If it has no slash, it is searched for in the directories of PATH, in order. shutil.which("fw_printenv") performs that same search and returns the full path, or None if the tool is absent, which is the clean way to check for a dependency before you need it. Be aware that some wrappers deliberately filter the environment: sudo, for example, strips variables such as LD_PRELOAD for security, so a child launched through it may not see what you set.
Going deeperHow CPython actually starts the child
Under subprocess.run on Linux, CPython normally forks and then execs the target program. It can take a posix_spawn fast path instead, but only when the options allow it: among other conditions, close_fds=False, no preexec_fn, no cwd, and an executable given with a directory component. With the defaults, you are on the fork-and-exec path. Two details follow. First, close_fds=True is the default, so the child inherits only the standard streams and any descriptors you pass explicitly; everything else is marked close-on-exec, which keeps your open sockets and serial ports out of the child. Second, if you watch with strace you will see clone or clone3, never a literal fork system call, because glibc implements fork with clone. The TimeoutExpired kill you saw earlier is SIGKILL sent to that child, followed by a wait to reap it.
logging: levels, handlers, and formatters
5 min readStart with the difference. print writes text to stdout, with no severity, no timestamp, and no way to route it. A a severity label on a message, from DEBUG up to CRITICAL, used to decide whether the message is worth keeping lets you keep detailed traces in a file while the console stays quiet, and change your mind later without editing code. The five standard levels, with their numeric values, are DEBUG (10), INFO (20), WARNING (30), ERROR (40), and CRITICAL (50). A message passes only if its level is at or above the threshold that is set.
The quickest setup is basicConfig, which configures the root logger:
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
log = logging.getLogger(__name__)
log.info("flashing %s", "fw.bin")
log.warning("boot count is %d", 3)
Two habits in that snippet matter. First, logging.getLogger(__name__) names the logger after the module, so in devkit/flash.py the logger is devkit.flash. Loggers form a hierarchy by dotted name, and a record propagates up to its ancestors' handlers, which is how one configuration covers a whole package. Second, pass values as arguments (log.info("flashing %s", name)) rather than building the string yourself; the formatting then happens only if the message will actually be emitted. One gotcha: basicConfig does nothing if the root logger already has handlers, which is why a second call in a test or a notebook appears to be ignored; pass force=True to reconfigure.
For a real tool you configure handlers explicitly. A an object attached to a logger that decides where a record goes, such as a stream, a file, or the network is the destination, and a formatter shapes the text. There are two gates, and both must pass: the logger's level filters first, then each handler's level filters again. A logger set to DEBUG with a handler set to WARNING emits only WARNING and above through that handler. This two-gate design is what lets one logger feed a verbose file and a quiet console at the same time:
import logging
from logging.handlers import RotatingFileHandler
log = logging.getLogger("devkit")
log.setLevel(logging.DEBUG)
console = logging.StreamHandler() # stderr by default
console.setLevel(logging.INFO)
console.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
fileh = RotatingFileHandler("devkit.log", maxBytes=1_000_000,
backupCount=5, encoding="utf-8")
fileh.setLevel(logging.DEBUG)
fileh.setFormatter(logging.Formatter(
"%(asctime)s %(levelname)s %(name)s: %(message)s"))
log.addHandler(console)
log.addHandler(fileh)
StreamHandler() writes to stderr, not stdout, which is the right default: your tool's real output stays on stdout for pipes, and diagnostics go to stderr. RotatingFileHandler solves the log that grows forever: when the file reaches maxBytes, it is renamed to devkit.log.1, older backups shift along, and a fresh file starts, keeping at most backupCount old files. TimedRotatingFileHandler rotates on a schedule instead, such as when="midnight". One caveat: rotation is designed for a single process; if several processes write one file, use a QueueHandler feeding one writer, or rotate externally.
For anything beyond a small script, put the configuration in data, not code. logging.config.dictConfig takes a dictionary, which you can load from JSON or build in one place:
import logging.config
logging.config.dictConfig({
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"plain": {"format": "%(levelname)s: %(message)s"},
"full": {"format": "%(asctime)s %(levelname)s %(name)s: %(message)s"},
},
"handlers": {
"console": {"class": "logging.StreamHandler",
"level": "INFO", "formatter": "plain"},
"file": {"class": "logging.handlers.RotatingFileHandler",
"level": "DEBUG", "formatter": "full",
"filename": "devkit.log",
"maxBytes": 1000000, "backupCount": 5},
},
"loggers": {
"devkit": {"level": "DEBUG", "handlers": ["console", "file"]},
},
})
disable_existing_loggers=False keeps loggers that modules already created at import time; the default True silences them, a surprise worth knowing. The older fileConfig reads an ini-style file and still works, but dictConfig is the current recommendation. And if you log before configuring anything, Python uses a last-resort handler that writes WARNING and above to stderr, so early messages are not silently lost.
Call: log.info creates a LogRecord with the message, level, time, and module.
Logger gate: the record passes only if its level meets the logger's effective level.
Handler gates: each attached handler checks the record against its own level.
Format: the handler's formatter renders the record into text.
Emit: the handler writes the text to its stream or file.
Propagate: unless stopped, the record also travels to ancestor loggers' handlers.
Logs outlive the run: they are copied, shipped to dashboards, and read by people who should not see credentials. Never log passwords, API tokens, private keys, or full connection strings with credentials. Log a fingerprint or a redacted form instead, and review what your DEBUG lines print before enabling them in production.
On a board, logging has a physical cost. Flash storage wears with every write, so a DEBUG-level file log on the device can shorten its life; keep verbose logs on the host, log to a tmpfs, or rate-limit. And if the root filesystem is read-only, point the file handler at a writable partition or drop the file handler entirely.
mmap and the edge cases of paths
5 min readA a region of a process's address space backed directly by a file, so reads and writes touch the file's pages through the kernel's page cache is what the mmap module exposes. Open the file in binary, then map it:
import mmap
with open("boot.log", "rb") as f:
with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
i = mm.find(b"kernel panic")
if i != -1:
print(mm[i:i + 200].decode("utf-8", errors="replace"))
last = mm.rfind(b"error")
Length 0 means the whole file. The mapping behaves like a bytes object: find, rfind, slicing, and indexing all work, and mm[i] is an integer byte. Nothing is read up front. The kernel loads a page only when you first touch it, through a page fault, and evicts pages under memory pressure. That makes mmap excellent for random access and repeated scans of a huge file: you pay only for the regions you actually read, and the page cache is shared with every other process mapping the same file. For a single sequential pass, chunked reads are just as good and simpler.
Find which region an address falls in below:
The lifetime rules are where people get burned. On POSIX, the mapping stays valid even after you close the file descriptor; the mapping holds its own reference to the file. But the mmap object itself must stay alive: after mm.close(), any access raises ValueError. An empty file cannot be mapped at all and raises ValueError immediately, so check the size first. ACCESS_READ is read-only; ACCESS_WRITE requires the file opened "r+b" and writes go through to the file; ACCESS_COPY gives you private copy-on-write changes that never reach the disk. On Windows the rules differ, notably that an empty file cannot be mapped and the file cannot be resized while mapped, so test on every platform you ship to.
Going deepermmap hazards: truncation and durability
Two failure modes deserve a place in your mental model. First, if another process truncates the file while you have it mapped, touching a page beyond the new end raises SIGBUS, which kills the process by default; a log rotator that truncates in place can do this to your reader, which is one more reason to prefer rename-based rotation. Second, writes through an ACCESS_WRITE mapping land in the page cache, not necessarily on storage; call mm.flush() to push them, and remember that a crash before the kernel writes back can lose them, exactly as with buffered file writes. The mapping is a view of the page cache, not a durability guarantee.
Paths have their own edges, and most come from the difference between path math and filesystem reality. PurePath and its subclasses PurePosixPath and PureWindowsPath do path computation with no filesystem access at all, which is what you want when a Linux build host computes paths for a Windows machine or vice versa. Path itself is the concrete, platform-appropriate class: PosixPath on Linux, WindowsPath on Windows. Case is the classic trap: Linux filesystems are case-sensitive, so Boot.log and boot.log are different files; Windows and the default macOS filesystem are case-insensitive, so they are the same file. PureWindowsPath even compares case-insensitively to match Windows semantics, while PurePosixPath compares exactly. A tool that globs *.LOG on Linux and finds nothing has met this trap. Symlinks are the second: resolve() expands them, is_symlink() detects them, and rglob refuses to descend through them by default, so a tree walk stays inside the tree you named. When you need the truth about a path, ask the filesystem with stat() rather than trusting the string.
Common mistakes and how to fix them
2 min readThe script hangs with no output. Cause: a piped child filled the 64 KiB pipe buffer while the parent called wait() or communicate() was never reached. Diagnose: ps shows the child sleeping, and strace -p on the parent shows it blocked in wait4 while the child is blocked in write. Fix: drain the pipes with communicate(), or iterate the stream as data arrives; never wait() first with unread pipes.
Mojibake or UnicodeDecodeError on a file that looks fine. Cause: text mode decoded with the wrong encoding, often the locale default on a CI machine. Diagnose: open the file in binary and look at the offending bytes; check what encoding produced them. Fix: pass encoding="utf-8" explicitly, or the encoding the file actually uses; use binary mode for data that is not text.
The command works in your terminal but raises FileNotFoundError in the script. Cause: the script runs with a different environment or working directory, often because env= replaced the environment and dropped PATH, or a cron job starts with a minimal PATH. Diagnose: print shutil.which(tool) and os.environ["PATH"] inside the script. Fix: start env from os.environ.copy(), use absolute program paths, or check with shutil.which first.
The tool failed but the script carried on. Cause: no check=True and no look at returncode. Diagnose: log result.returncode and result.stderr after every call. Fix: check=True plus except subprocess.CalledProcessError, and branch on the code when different failures need different responses.
A filename turned into a command. Cause: shell=True with an f-string that interpolated untrusted text. Diagnose: search the codebase for shell=True and trace where every interpolated value comes from. Fix: pass a list; if a shell is unavoidable, shlex.quote every value.
Log lines are missing. Cause: the logger's level or the handler's level filters them out, or basicConfig ran after handlers already existed and did nothing. Diagnose: print log.getEffectiveLevel() and each handler's level. Fix: set both gates deliberately, and use force=True when reconfiguring.
Scratch files pile up in /tmp. Cause: NamedTemporaryFile(delete=False) without cleanup, or a crash before the cleanup line. Diagnose: list the prefix your tool uses. Fix: prefer TemporaryDirectory, whose cleanup runs even on exceptions, and keep scratch on the right filesystem with dir=.
Try it
1 min readThe lab has three parts that build on each other. First, write a pathlib script that walks a directory tree with rglob, classifies each file with stat, and prints a report, replacing a find plus awk one-liner. Second, run an external tool with subprocess.run, capture stdout and stderr, branch on the return code, and handle a timeout cleanly. Third, add level-aware logging with a console handler and a rotating file handler to that same script, then watch what each level gate lets through. The practice pack drills the same skills with shorter questions, including the deadlock and encoding traps. When you finish, the next lesson takes these tools to the hardware: serial ports with pyserial, SSH automation, and sockets.
Key takeaways
- pathlib turns paths into objects: / joins, glob finds, stat inspects, and read_text or write_text does the I/O in one call.
- with open(...) closes the file even when the block raises; always pass encoding= for text files.
- tempfile creates files nobody can predict or race; write-temp-then-os.replace makes an update atomic.
- Pass subprocess arguments as a list; shell=True hands your string to a shell and invites injection.
- A Linux pipe holds 64 KiB by default; wait() before draining stdout can deadlock, so use communicate().
- env= replaces the whole environment instead of merging, so start from os.environ.copy().
- logging gives you levels, timestamps, and several destinations at once; print gives you none of them.
- Bash still wins for quick pipes and one-liners; Python wins when logic, errors, and structured data appear.
Reflect
- Explain to a colleague, in two sentences: build and manipulate paths with pathlib: / joining, glob, stat, and the read and write helpers.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Official docspathlib: Object-oriented filesystem paths
- Official docssubprocess: Subprocess management
- Official docslogging: Logging facility for Python
- Official docslogging.config: Logging configuration
- Official docstempfile: Generate temporary files and directories
- Official docsshutil: High-level file operations
- Official docsmmap: Memory-mapped file support
- Official docsshlex: Simple lexical analysis
- SpecificationPEP 428: The pathlib module
Official documentation first. Links checked when the lesson was published.