Why an embedded product is built in layers
2 min readThere is no special "embedded" version of Linux. The same mainline kernel that runs a server also runs a smart thermostat; the difference is which pieces you ship and how you glue them together. What makes a product embedded is that the software exists to serve one dedicated job on fixed, resource-limited hardware, often for years without a human at the keyboard.
A the part of the operating system that runs in privileged mode and owns the hardware never boots straight into your application. Between the raw hardware and the finished feature there is always a chain of responsibility: a a small program that initialises memory and hardware and loads the kernel, the kernel, an init program, a set of system services, your application services, and finally the user-facing programs. Each layer hands control to the next along a well-defined seam.
Engineers should care about this structure because the seams are where real products fail. A missing shared library, a service that starts before its database, a configuration file written to a read-only partition, a secret leaked into a process environment: every one of these is a boundary problem, not an algorithm problem. When you understand who owns each layer and what contract connects them, you can reason about the whole device instead of guessing.
A boundary is a contract with three parts: an interface (the exact calls or files the layer exposes), a version (so both sides change in step), and an owner (the one team allowed to edit that layer). When any of the three is missing, the boundary silently dissolves and two layers start to depend on each other's internals, which is how a device becomes impossible to update.
How the layered stack evolved
2 min readEarly Unix systems started services with a single init program that ran shell scripts in order. This model, the first user-space process, PID 1, which starts and supervises everything else from System V in the mid-1980s, used numbered runlevels and /etc/init.d scripts, and it worked when a machine ran a handful of daemons in series.
Embedded Linux forced two pressures onto that model. First, storage was tiny. Bruce Perens started BusyBox in 1996 for the Debian installer so it could boot from a 1.44 MB floppy; BusyBox packs hundreds of utilities into one binary dispatched by the name it is called as, and it became the heart of embedded user space ever since. Second, boot time and parallelism mattered. PowerPC boot firmware PPCBoot (2000) became U-Boot (2002), giving boards a scriptable, network-capable bootloader. In 2010 Lennart Poettering and Kay Sievers released systemd, which replaced sequential shell scripts with declarative units, explicit dependencies, parallel startup, and built-in supervision.
Alongside it, the daemontools/runit tradition (Gerrit Pape's runit, and later s6) kept the Unix idea that a small supervisor should own each service, run it in the foreground, and restart it when it dies. Meanwhile the filesystem merged /bin into /usr/bin, /var/run became the /run tmpfs, and control groups moved the kernel toward per-service resource accounting. Each of these is still a live choice in new products, which is why this module treats layering as a design decision, not a fixed recipe.
Explore the turning points below:
The product layer model, top to bottom
3 min readThe six layers, and what each is responsible for:
- Bootloader. Initialises DRAM and essential clocks, finds the kernel and device tree, and jumps into them. U-Boot is the common choice.
- Kernel. Owns the hardware: device drivers, process scheduling, memory, filesystems, and the system call interface. It runs privileged and provides services to everything above.
- Init / process supervisor. The first user-space process (PID 1). It mounts the early filesystems, then starts and watches the daemons. systemd, BusyBox init, and runit all fill this slot.
- System services. The shared plumbing: the D-Bus message bus, the logger, the network manager,
udevfor device nodes. These are platform, not product. - Application services. Your product's own daemons: sensor ingestion, storage, cloud uplink. Each is a normal process supervised by layer 3.
- User apps. Anything a person or an operator touches directly: a local UI, a CLI tool, a web page served from the device.
The boundary matters because responsibilities are not interchangeable. A driver bug is fixed in the kernel layer without touching the app; a cloud-protocol change is fixed in the application layer without rebuilding the platform. When a team blurs these lines, a "small" change ripples across the whole image and every test.
Boot ROM: the CPU runs masked ROM that loads the first-stage loader from flash
Bootloader: U-Boot initialises DRAM and clocks, then loads the kernel and device tree
Kernel: initialises drivers and schedulers, mounts the root filesystem, then executes init as PID 1
Init: the supervisor mounts /proc and /sys and starts the base target
System services: D-Bus, logging, and networking come up, in parallel where dependencies allow
Application services: your daemons start once their dependencies are satisfied
First output: the app opens its UI or publishes its first message
Step through the handover below:
Where this model runs today is everywhere: a home router is layers 1-5 with no UI, a car head unit is the full stack with a graphics-heavy layer 6, and a sensor gateway is a thin stack where layers 4 and 5 dominate the code.
Application versus platform: the kernel-user seam
3 min readThe split is: business logic, user interface, and protocol handling belong in the application layer; device drivers, middleware, and the kernel belong below. The seam is the the controlled entry point a user-space program uses to ask the kernel to do something boundary plus the two pseudo-filesystems and device files that sit on top of it.
Applications read hardware state from sysfs. For example, a a virtual filesystem that exposes the kernel's device model as files and directories attribute gives you one value per read:
cat /sys/class/leds/led0/brightness # read current level
echo 0 > /sys/class/leds/led0/brightness # turn the LED off
A a special file under /dev that routes reads and writes to a specific driver is how you talk to character devices. The I2C bus is a good example: the kernel provides a generic /dev/i2c-N driver, and your code uses an a system call for device-specific control operations that plain read and write cannot express to select a slave, then reads and writes bytes:
int f = open("/dev/i2c-0", O_RDWR);
ioctl(f, I2C_SLAVE, 0x50); /* address the EEPROM on the bus */
char buf[4] = {0, 0};
write(f, buf, 2); /* point at register 0 */
read(f, buf, 4); /* read four bytes back */
The same pattern holds for SPI (/dev/spidevX.Y) and, on modern kernels, for GPIO through the gpiochip character devices driven by libgpiod and its gpioset/gpioget tools. The older /sys/class/gpio/export interface is deprecated; new work should use the character device.
None of this works unless the device nodes actually exist, and on a modern system they are not created by hand with mknod. The kernel mounts devtmpfs on /dev and fills it as drivers register their devices, while udev (or BusyBox's lighter mdev) reacts to each hotplug event, applies naming and permission rules, and loads the matching module. That is why a newly wired sensor shows up under /dev or /sys/class with the right owner and no shell script involved.
Going deeperWhy user space must not touch registers directly
A SoC exposes its peripherals through memory-mapped registers, and /dev/mem would let a root process poke them. Doing this from an application is wrong for reasons that survive contact with hardware: the kernel driver arbitrates a shared bus so two processes cannot talk over each other; it handles interrupts, DMA, and cache coherency that a bare store to a register would corrupt; and it enforces permissions and power management that a raw write cannot respect. Kernel code that touches registers uses readl()/writel() helpers rather than C volatile, precisely because memory ordering and I/O barriers matter. An application that maps registers is bypassing every one of these guarantees to save a few lines of code, and it will fail in the field in ways that are hard to reproduce.
One monolith or many services
2 min readA monolithic application is simpler to start with: one binary, one main loop, no inter-process communication to debug. It starts to hurt as the product grows, for four concrete reasons. Isolation: one crash brings down the whole feature set, while separate processes contain the blast radius. Restart: a supervisor can restart one failed daemon without touching the others. Testability: a service with a small interface can be tested and mocked on its own. Ownership: separate processes map cleanly onto separate teams, separate update cadences, and separate resource budgets.
The worked example for this module is the SensorNode gateway. As one monolith it would read sensors, store history, upload to the cloud, draw a local screen, and update itself, all in one binary. Decomposed, it becomes five application services that each sit above the platform layers:
- sensor-ingest samples the I2C/SPI sensors and publishes normalised readings.
- storage owns a local SQLite database and answers read/write requests.
- cloud-uplink buffers data and publishes it to an MQTT broker.
- local-ui renders the status screen and accepts operator input.
- ota-agent downloads, verifies, and stages firmware updates.
The boundary between these services is deliberately thin: they talk over D-Bus for local control-plane messages and pass payloads through a small message layer rather than calling into each other's code. The app layer as a whole sits on the platform the way every other product does, through sysfs, device nodes, and the C library, so the SensorNode's five daemons are all layer 5 and layer 6, never layer 4.
Service lifecycle with systemd
3 min readA a systemd configuration file that describes a service, target, timer, socket, or mount is a text file. Here is a minimal service for one SensorNode daemon:
[Unit]
Description=SensorNode storage service
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/sensornode-storage
Restart=on-failure
RestartSec=2
[Install]
WantedBy=multi-user.target
Type=simple means the ExecStart process is the service; forking, oneshot, and notify cover daemons that double-fork, run once, or signal readiness. Restart=on-failure restarts only when the process exits with a failure, and RestartSec=2 inserts a two-second pause to avoid a crash loop; always, on-abnormal, and on-watchdog are the other common policies.
Dependencies live in the [Unit] section. Requires= starts a unit this one needs and fails this one if that fails; Wants= starts it but tolerates failure; After=/Before= only impose ordering, not activation. WantedBy=multi-user.target plus systemctl enable drops a symlink into multi-user.target.wants/, which is what actually schedules the service at boot.
Beyond services, three unit kinds matter for products. Socket activation (foo.socket) has systemd create and listen on the socket and start the service on first connection, so idle services cost nothing. Timer units replace cron with calendar or monotonic schedules. Template units (name@.service) let one file serve many instances, with %i expanding to the instance name, as getty@tty1 does. A separate user manager (systemd --user, backed by user@<uid>.service) supervises one logged-in user's own services outside the system PID 1 tree, which keeps a signed-in operator's graphical session and its daemons cleanly separated from the product services.
Kernel: runs /sbin/init, which is a symlink to systemd, as PID 1
Early setup: systemd mounts /proc, /sys, /dev and reads its unit files
Default target: resolves default.target to multi-user.target or graphical.target
Dependency graph: walks Requires/Wants/After edges to collect the full set of units
Ordering: computes a start order that honours every After/Before constraint
Parallel start: launches independent services concurrently within that order
Steady state: watches each service, applying its Restart= policy on exit
Going deeperResource control and the watchdog, in practice
systemd places each service in a cgroup (the unified cgroup v2 hierarchy on modern kernels), so limits are per service. CPUQuota=200%, MemoryMax=256M, and IOWeight= in the [Service] section bound what one daemon may consume. A hardware a timer that reboots the system unless software keeps resetting it in time is handled two ways: RuntimeWatchdogSec= in /etc/systemd/system.conf makes systemd itself kick /dev/watchdog, and per-service WatchdogSec=30s expects the daemon to call sd_notify(0, "WATCHDOG=1") within the window; if it stalls, systemd restarts it. StartLimitBurst= and StartLimitIntervalSec= cap restarts, and StartLimitAction=reboot-force turns a crash-looping service into a full reset rather than an endless restart.
Choosing an init and supervisor
3 min readBusyBox init reads /etc/inittab, a list of lines in the form id::action:program. The sysinit action runs mount commands once at boot; respawn restarts a program whenever it exits; askfirst waits for a keypress before giving a shell on the console. It is tiny and deterministic, and Buildroot's rcS/rcK scripts give it a simple start/kill convention. What it does not do is supervise: a respawn daemon is restarted, but there are no per-service logs, no dependency graph, and no rate limits.
System V init extends the same idea with runlevels (0-6 plus S) and /etc/rcN.d directories of S##/K## symlinks. It is more modular than BusyBox init but still has no supervision and no parallelism, and it is now mostly a compatibility layer.
systemd adds declarative units, a real dependency graph, parallel startup, per-service logging, resource control, and supervision, at the cost of size (roughly 10 MiB for the core with udevd and journald) and a hard requirement on glibc.
runit (and its sibling s6) takes the opposite design: runsvdir watches a directory of services, and one runsv process supervises each. A service is just an executable run script that must run in the foreground:
#!/bin/sh
exec /usr/sbin/sshd -D
runsv restarts the service when run exits, a finish script runs on shutdown, a check script expresses readiness, and sv (sv up, sv down, sv once) controls it. Products pick runit for deterministic boot, a small footprint, and supervision without systemd's machinery; services start concurrently, so a dependent daemon simply retries until its prerequisite appears. s6 goes further and models the whole system as a formal supervision tree (s6-svscan), which is why some teams choose it over both systemd and runit for determinism and a small attack surface.
Logging follows the supervisor. systemd's journald captures every service's output in a structured, queryable journal. Traditional syslogd writes plain text lines such as /var/log/messages, while kernel messages arrive separately via klogd (or journald reading /dev/kmsg directly). runit's svlogd gives each service its own rotated log directory. On flash, all of these must be reined in: log to tmpfs or a ring buffer, cap the size, and rotate, because thousands of small writes wear flash and can fill a small partition.
Going deepersvlogd rotation, exactly
svlogd appends to a file named current and rotates it when it reaches a size set by sSIZE in the directory's config file (default 1000000 bytes), keeping up to nNUM rotated files (default 10). On rotation it closes current, renames it to a timestamped name, and starts a fresh file. The -t/-tt/-ttt flags stamp each line with TAI64N or human-readable time. Because each service's logger is its own process piped from the service's stdout, one noisy daemon can never flood another's log file, and a crashed logger does not take the service down.
Libraries, the dynamic linker, and middleware
4 min readStatic linking copies the library code into the executable at build time. gcc -static app.c -o app pulls from .a archives, so the binary is large but self-contained; it needs no runtime loader and no libraries on the target. Dynamic linking builds against .so shared objects, and the binary records which ones it needs as NEEDED entries plus the path of the dynamic loader, ld.so, in the ELF interpreter field:
readelf -d app | grep -E 'NEEDED|interpreter'
# 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
# [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
ld.so resolves those names at startup using /etc/ld.so.conf, the cache built by ldconfig, and the default /lib and /usr/lib. A library's the ABI name of a shared library, encoding the interface version that compatible programs link against is what makes updates safe: libfoo.so.8 is the interface, and libfoo.so.8.2.2 is a specific release behind it, so a bugfix can ship without relinking every program. The resolution itself is a fixed sequence:
1Map the interpreter
The kernel's exec path reads the ELF program header, finds the PT_INTERP entry, and maps the named loader (ld.so) into the new process instead of jumping straight to main.
2Read the needed libraries
ld.so parses the program's DT_NEEDED entries to build the list of libraries that must be present before the program can run.
3Assemble the search path
It builds the lookup order: DT_RPATH, then LD_LIBRARY_PATH (unless secure execution), then DT_RUNPATH, then the ldconfig cache, then the trusted defaults /lib and /usr/lib.
4Load and resolve recursively
Each needed library is mapped in, and its own DT_NEEDED entries are loaded the same way, forming a tree of dependencies rather than a flat list.
5Relocate and enter
Relocations are applied, symbols are resolved (at load time when the binary is built -z now), and control transfers to the program's entry point.
Going deeperRPATH, RUNPATH, and why LD_LIBRARY_PATH sometimes does nothing
An executable can record its own search paths in the DT_RPATH or DT_RUNPATH dynamic tags (-rpath). The order matters: DT_RPATH is searched before LD_LIBRARY_PATH, while DT_RUNPATH (what modern toolchains emit) is searched after it and only for the object's direct dependencies. Ubuntu's toolchain also links binaries with immediate binding (-z now, BIND_NOW), so symbol resolution happens at load time rather than lazily. For security, the loader enters secure-execution mode for set-user-ID programs and ignores LD_LIBRARY_PATH and RUNPATH entirely, which is also why sudo strips LD_* variables: you cannot inject a library into a privileged process by pointing the environment at it.
The choice is not neutral. Static linking increases footprint when many programs would share a library, complicates security updates (a libc fix requires relinking everything), and interacts with licences: an LGPL library allows dynamic linking without the copyleft spreading to your application, while static linking pulls the library's code into your binary and brings licence obligations with it. Dynamic linking shrinks storage and makes updates cheap but adds the whole loader-and-soname discipline, plus plugins loaded at runtime with dlopen.
Above the C library sits the middleware a product reuses rather than writes: D-Bus for local IPC between services, MQTT for publish/subscribe telemetry to a broker, SQLite for a single-file embedded database, Qt or LVGL for the UI (LVGL for small, resource-limited screens), GStreamer for multimedia pipelines, and an OTA agent such as RAUC or SWUpdate for in-field updates. D-Bus runs one system bus per device (for system services) and one session bus per user; MQTT decouples publishers from subscribers through a broker and offers three quality-of-service levels for delivery guarantees. These are platform-adjacent libraries and daemons: your application services call them, but they are not your product logic.
Filesystem layout, read-only roots, and configuration ownership
3 min readThe Filesystem Hierarchy Standard divides the tree by mutability. /etc is configuration; /var is persistent, runtime-writable data; /run is volatile runtime state such as PID files and sockets, backed by tmpfs; /tmp is scratch space that may be cleared. Modern distributions also merge /bin and /sbin into /usr/bin and /usr/sbin, and /var/run is now a symlink to /run.
Embedded products take this one step further and make the root filesystem read-only, mounting it ro or shipping it on a read-only filesystem such as SquashFS. This stops accidental writes and corruption, but a few well-known paths still expect to be writable: /etc/resolv.conf becomes a symlink into a tmpfs, /var/log is mounted on tmpfs so logs are volatile, and /var/lib may be copied to tmpfs and bind-mounted if an application insists on writing there. Buildroot maps /var/cache, /var/log, and friends onto /tmp; Yocto uses a /var/volatile tmpfs with symlinks. The writable, persistent data lives on a separate partition, usually /data or /var, so an OTA can replace the whole read-only image while user data survives. That persistent partition is the one thing an update must never touch, and it is sized for years of accumulated data rather than for the current firmware.
This layout answers the question of who owns configuration. The platform owns base configuration: network setup, systemd units, and device policy live in /etc and ship with the image. Each application service owns its own slice, conventionally /etc/<app>/, and must not edit another service's files. Secrets need the tightest ownership of all. A credential in the environment is visible in /proc/<pid>/environ and leaks into every child process and core dump; a credential in a file with mode 0600, owned by the service's own user, is read once at startup and stays put. systemd's LoadCredential= goes further and hands the secret to the service on an unlinked, memory-backed file, so it never lands in the environment at all.
Try the permission bits on a credential file:
Common mistakes and the boot-time budget
3 min readFour mistakes recur in real products, each with a signature, a cause, and a fix:
- Missing dynamic loader or library. Symptom: the app dies at exec with
cannot execute: required file not found. Cause: the ELF interpreter path or aNEEDEDlibrary is absent on the target. Diagnose withreadelf -l app | grep interpreterandreadelf -d app | grep NEEDED, then add the missing files to/libor/usr/lib. Fix: pin the libraries in the image and add a boot-time check. - Wrong start order. Symptom: a service crashes and restarts a few times right after boot, then settles. Cause: it starts before its dependency (D-Bus, network, a device) is ready. Diagnose with
systemctl statusor the runitcheckscript; fix with an explicitAfter=/Requires=edge or a readiness check rather than a fixed sleep. - Logs filling flash. Symptom: the device boots to a full, read-only filesystem. Cause: unbounded logging to a persistent partition. Fix: route logs to tmpfs, cap sizes, and rotate, exactly as the previous section describes.
- Secrets in the environment or in code. Symptom: a credential is visible in a process listing or a leaked core dump. Fix: move it to a
0600file owned by the service user, or a systemd credential.
The boot-time budget is where layers 3 and 4 spend the most time. Measure with systemd-analyze and systemd-analyze blame to see which units cost the most, and systemd-analyze critical-chain to see the longest dependency path. The application layer itself usually contributes little; its dependencies do, so the levers are deferring non-critical services (Type=idle starts a unit only when the CPU is otherwise free), not blocking the first output on network or on a slow peripheral, and moving unrelated work after the UI is up. Time-to-first-app-output is a product metric: whatever the screen or the first MQTT message waits on is, by definition, on the critical path.
Interface boundaries are contracts, and contracts have versions. The soname is the platform's ABI versioning; your services need the same discipline between layers, or a change to one daemon's message format silently breaks the next one. The cost of coupling is paid in every future change, so the cheapest time to draw a boundary and pin its version is before the first integration, not after.
Try it
1 min readOpen the lab for this lesson and boot your target, then run pstree -a and systemctl list-units --type=service (or sv status /etc/sv/* under runit) and map every process you see onto the six layers, drawing the resulting diagram. Write a systemd unit or a runit service for a small program that exits after a few seconds, set Restart=on-failure (or respawn/runsv) and dedicated logging, and confirm it restarts and logs separately. Finally, decompose the SensorNode gateway on paper into its five services, list the IPC between them, and mark the exact line where the application layer ends and the platform begins. The practice pack has the step-by-step commands for each of these.
Key takeaways
- A product is a stack: bootloader, kernel, init, system services, application services, user apps.
- Each layer owns one responsibility; the boundary between layers is a contract, not a suggestion.
- Applications talk to hardware through sysfs, device nodes and ioctl, never through raw registers.
- Split a product into services for isolation, restart, testability and clear ownership.
- systemd, BusyBox init and runit differ in supervision, logging, footprint and boot time.
- Static linking copies library code in; dynamic linking shares it and changes your update story.
- A read-only root with a writable /var and tmpfs protects flash and survives power loss.
- Secrets belong in files with tight permissions, not in the environment or in source code.
Reflect
- Explain to a colleague, in two sentences: identify each layer of an embedded product and state which responsibility lives there
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- SpecificationFilesystem Hierarchy Standard
- Man pagesystemd.service(5)
- Man pagesystemd.unit(5)
- Man pageld.so(8)
- Man pagesysfs(5)
- Official docsLinux kernel GPIO documentation
- SpecificationD-Bus Specification
- Official docsrunit documentation
Official documentation first. Links checked when the lesson was published.