all
Stage 09

systemd

Master systemd for embedded and server Linux — unit files, service/target/socket/timer units, systemctl commands, journald logging, socket activation, boot time optimization with systemd-analyze, and sd-boot.

BusyBox Root Filesystem SysVinit & Runlevels
12 min read
66129 chars

Unit Files

{:.gc-basic}

Basic

systemd manages the system through unit files — declarative INI-format text files that describe services, devices, mounts, timers, sockets, and more. systemd reads unit files from several locations in order of precedence:

Path Purpose
/etc/systemd/system/ Local admin overrides — highest precedence
/run/systemd/system/ Runtime-generated units
/lib/systemd/system/ Distribution-provided units — do not edit
/usr/lib/systemd/system/ Vendor units (same as above on some distros)

Files in /etc/systemd/system/ shadow those with the same name in /lib/systemd/system/. Use systemctl edit servicename to create a drop-in override without touching the original.

Unit Types

Extension Manages
.service A daemon or one-shot process
.target A synchronisation point / group of units
.socket A socket for socket-activated services
.timer A calendar or monotonic timer
.mount A filesystem mount point
.automount On-demand (lazy) mount
.device A kernel device (from udev)
.path A filesystem path watch
.scope Externally created processes (e.g., login sessions)
.slice cgroup hierarchy node for resource control

Anatomy of a .service File

# /etc/systemd/system/myapp.service

[Unit]
Description=My Embedded Application
Documentation=https://example.com/myapp/docs
# Start after networking is up
After=network-online.target
# network-online must be active, not just started
Wants=network-online.target
# If postgresql fails to start, this unit will not start either
Requires=postgresql.service

[Service]
# Type tells systemd how to track readiness:
#   simple   — ExecStart is the main process (default)
#   forking  — process double-forks; systemd tracks the parent
#   notify   — process calls sd_notify(READY=1) when ready
#   oneshot  — runs to completion (no persistent process)
#   dbus     — service registers a D-Bus name when ready
Type=notify

# Run as a non-root user
User=myapp
Group=myapp

WorkingDirectory=/opt/myapp

# The main executable
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.toml

# Graceful reload without restart (send SIGHUP)
ExecReload=/bin/kill -HUP $MAINPID

# Pre-start script (runs as root before User= drop)
ExecStartPre=/opt/myapp/bin/preflight-check.sh

# Load environment variables from a file
EnvironmentFile=-/etc/myapp/myapp.env

# Restart policy
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=60s
StartLimitBurst=3

# Resource limits
LimitNOFILE=65536
LimitMEMLOCK=infinity

# Sandboxing (recommended for production)
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

# Standard output/error go to journald
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
# This unit starts when multi-user.target is reached
WantedBy=multi-user.target

[Unit] Dependency Keywords

Keyword Meaning
Requires= Hard dependency — if the listed unit fails, this unit fails too
Wants= Soft dependency — listed unit is started alongside, failure is tolerated
After= Ordering — this unit starts after the listed unit is active
Before= Ordering — this unit starts before the listed unit
BindsTo= Like Requires= but also stops this unit if the dependency stops
Conflicts= This unit cannot run simultaneously with the listed unit
PartOf= Stops/restarts with the listed unit, but not started by it

After= and Requires= are independent — Requires=foo.service does not imply After=foo.service. Always set both if you need ordering AND hard dependency.

systemctl Commands

{:.gc-basic}

Basic

Service Lifecycle

# Start / stop / restart
systemctl start   myapp.service
systemctl stop    myapp.service
systemctl restart myapp.service

# Reload configuration without full restart (sends ExecReload signal)
systemctl reload  myapp.service

# Restart only if already running
systemctl try-restart myapp.service

# Detailed status including last log lines
systemctl status  myapp.service

Example output of systemctl status:

● myapp.service - My Embedded Application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2026-03-13 10:42:17 UTC; 3min ago
    Process: 1042 ExecStartPre=/opt/myapp/bin/preflight-check.sh (code=exited, status=0/SUCCESS)
   Main PID: 1057 (myapp)
     CGroup: /system.slice/myapp.service
             └─1057 /opt/myapp/bin/myapp --config /etc/myapp/config.toml

Enable / Disable (Boot Persistence)

# Create symlink in .wants/ directory of the target — persists across reboots
systemctl enable  myapp.service

# Enable and start in one command
systemctl enable --now myapp.service

# Remove symlink — service no longer starts at boot
systemctl disable myapp.service

# Disable and stop in one command
systemctl disable --now myapp.service

Inspecting Units

# List all active units
systemctl list-units

# List all installed unit files and their state
systemctl list-unit-files

# Filter by type
systemctl list-units --type=service
systemctl list-units --type=timer

# Quick boolean checks (exit code 0 = yes)
systemctl is-active  myapp.service
systemctl is-enabled myapp.service
systemctl is-failed  myapp.service

# Show all failed units
systemctl --failed

Configuration Reload

# Always run after editing or adding a unit file
# Tells systemd to re-read unit files from disk without restarting anything
systemctl daemon-reload

Viewing Logs

# Follow logs for a specific service
journalctl -u myapp.service -f

# Show logs since last boot
journalctl -u myapp.service -b

# Show last 50 lines
journalctl -u myapp.service -n 50

# Show only errors and above
journalctl -u myapp.service -p err

Targets (Replacing Runlevels)

{:.gc-mid}

Intermediate

systemd targets are named synchronization points that group units together, replacing the numeric runlevels of SysVinit.

systemd Target SysV Runlevel Description
poweroff.target 0 Shut down and power off
rescue.target 1 Single-user mode, minimal services
multi-user.target 2, 3, 4 Multi-user, network, no GUI
graphical.target 5 Multi-user + display manager
reboot.target 6 Reboot
emergency.target — Bare minimum, root shell, no mounts

Dependency Chain (simplified)

sysinit.target
  └── local-fs.target
  └── swap.target
  └── cryptsetup.target
        └── basic.target
              └── multi-user.target
                    └── graphical.target

Target Management

# Switch to a different target right now (like telinit)
systemctl isolate rescue.target
systemctl isolate multi-user.target

# Set the default target (persists across reboots)
systemctl set-default multi-user.target
systemctl get-default

# See all target dependencies
systemctl list-dependencies multi-user.target

Masking Units

Masking completely prevents a unit from being started, even as a dependency:

# Mask — symlinks the unit to /dev/null
systemctl mask bluetooth.service

# Unmask
systemctl unmask bluetooth.service

This is stronger than disable — even systemctl start will refuse a masked unit.

Minimal Embedded Target

For resource-constrained embedded systems, create a custom target that skips unnecessary services:

# /etc/systemd/system/embedded.target
[Unit]
Description=Embedded Minimal Target
Requires=sysinit.target
After=sysinit.target
AllowIsolate=yes

Timers

{:.gc-mid}

Intermediate

systemd timers replace cron with a tighter integration into the service manager — missed runs can be caught up, timers are logged in journald, and they respect service dependencies.

Timer Unit Anatomy

A timer needs a matching .service unit with the same base name:

# /etc/systemd/system/cleanup.timer
[Unit]
Description=Daily /tmp Cleanup Timer
Requires=cleanup.service

[Timer]
# Run at 03:15 every day
OnCalendar=*-*-* 03:15:00

# If the system was off at the scheduled time, run it soon after next boot
Persistent=true

# Randomize start time by up to 5 minutes (avoids thundering herd)
RandomizedDelaySec=5min

[Install]
WantedBy=timers.target

# /etc/systemd/system/cleanup.service
[Unit]
Description=Daily /tmp Cleanup

[Service]
Type=oneshot
ExecStart=/bin/find /tmp -maxdepth 1 -mtime +7 -delete
ExecStart=/bin/find /var/tmp -maxdepth 1 -mtime +30 -delete

Timer Activation Keywords

Keyword Triggers
OnCalendar= Absolute time (like cron). Format: DayOfWeek Year-Month-Day Hour:Min:Sec
OnBootSec= Relative to system boot (e.g., OnBootSec=10min)
OnActiveSec= Relative to when the timer unit was activated
OnUnitActiveSec= Relative to the last time the matching service ran
OnUnitInactiveSec= Relative to when the service last became inactive

Common Calendar Expressions

OnCalendar=daily           # equivalent to *-*-* 00:00:00
OnCalendar=weekly          # Monday *-*-* 00:00:00
OnCalendar=monthly         # *-*-01 00:00:00
OnCalendar=hourly          # *-*-* *:00:00
OnCalendar=Mon,Thu 12:00   # Monday and Thursday at noon
OnCalendar=*-*-* 02:00~03:00/30  # random between 02:00 and 03:00 every 30 min

# Verify a calendar expression
systemd-analyze calendar "Mon,Thu 12:00"

# List all active timers with next/last run times
systemctl list-timers --all

Socket Activation

{:.gc-mid}

Intermediate

Socket activation lets systemd create and hold a socket open before the backing service starts. The service is launched on first connection, and connections received while the service is starting are queued in the kernel’s socket buffer — zero connections are dropped.

How It Works

  1. systemd reads the .socket unit and calls socket() + bind() + listen().
  2. The socket file descriptor is passed to the service process via SD_LISTEN_FDS_START (fd 3).
  3. The service calls sd_listen_fds() to discover inherited sockets instead of calling socket() itself.

Socket Unit

# /etc/systemd/system/myserver.socket
[Unit]
Description=My Server Socket

[Socket]
# TCP socket on port 8080
ListenStream=0.0.0.0:8080

# OR a Unix domain socket
# ListenStream=/run/myserver/myserver.sock

# Pass socket to service (default yes)
Accept=no

[Install]
WantedBy=sockets.target

Matching Service Unit

# /etc/systemd/system/myserver.service
[Unit]
Description=My Server (socket-activated)
Requires=myserver.socket
After=myserver.socket

[Service]
Type=notify
ExecStart=/usr/sbin/myserver
# Socket is inherited on fd 3; service must not call bind()
StandardInput=socket

Benefits for Embedded Systems

  • Lazy start: services only consume RAM when first needed
  • Zero-downtime restart: during restart, the socket stays open and the kernel buffers connections
  • Parallel boot: all sockets are created before any service starts, eliminating dependency ordering for socket consumers

Minimal C Example — Reading the Activated Socket

#include <systemd/sd-daemon.h>
#include <unistd.h>

int main(void) {
    int n = sd_listen_fds(0);
    if (n < 1) {
        /* Fallback: create socket manually for direct invocation */
        /* ... */
    }
    int fd = SD_LISTEN_FDS_START;  /* fd 3 */
    /* Use fd for accept() or recv() */
}

Boot Optimization

{:.gc-adv}

Advanced

Measuring Boot Time

# Total time broken into firmware, loader, kernel, userspace
systemd-analyze time

# Example output:
# Startup finished in 1.203s (kernel) + 3.847s (userspace) = 5.050s

# List every service sorted by startup time
systemd-analyze blame

# Show the critical path through the dependency graph
systemd-analyze critical-chain

# Generate a SVG flame graph of the boot
systemd-analyze plot > boot.svg

Reducing Boot Time on Embedded Systems

# 1. Disable unneeded services
systemctl disable ModemManager.service
systemctl disable avahi-daemon.service
systemctl disable cups.service
systemctl disable bluetooth.service

# 2. Mask units that would otherwise be pulled in as dependencies
systemctl mask systemd-resolved.service
systemctl mask systemd-timesyncd.service   # if no NTP needed

# 3. Check for slow services
systemd-analyze blame | head -20

Type=notify with sd_notify()

When a service sets Type=notify, systemd waits for the process to call sd_notify(0, "READY=1") before considering it started. This avoids the race condition where systemd marks a service “active” before it has finished initialising:

#include <systemd/sd-daemon.h>

int main(void) {
    /* ... complete all initialisation ... */
    sd_notify(0, "READY=1");        /* Tell systemd we are ready */
    sd_notify(0, "STATUS=Listening on port 8080");  /* Optional status string */

    /* Main event loop */
    for (;;) {
        /* ... */

        /* Watchdog — must be sent within WatchdogSec= interval */
        sd_notify(0, "WATCHDOG=1");
    }
}

[Service]
Type=notify
WatchdogSec=30s
NotifyAccess=main   # or "all" if using threads/forks

Minimal Embedded systemd Configuration

For embedded targets (e.g., a Yocto or Buildroot system), disable all non-essential features at compile time:

# meson_options.txt flags for minimal systemd build
-Dnss-myhostname=false
-Dnss-resolve=false
-Dresolve=false
-Dlogind=false
-Dmachined=false
-Dhostnamed=false
-Dtimedated=false
-Dlocaled=false
-Dcoredump=false
-Dpolkit=false
-Daudit=false
-Dselinux=false

journald

{:.gc-adv}

Advanced

systemd-journald collects log entries from kernel messages (/dev/kmsg), syslog socket, and service stdout/stderr. Entries are stored in a structured binary format with indexed fields.

Storage Paths

Path Condition
/run/log/journal/ Volatile (RAM) — exists when /var/log/journal/ does not exist. Lost on reboot.
/var/log/journal/ Persistent — created automatically if /var/log/journal/ exists (mkdir /var/log/journal)

journalctl Query Reference

# All logs from current boot
journalctl -b

# Logs from previous boot
journalctl -b -1

# Kernel messages only (dmesg equivalent)
journalctl -k

# Filter by priority (emerg=0, alert=1, crit=2, err=3, warn=4, notice=5, info=6, debug=7)
journalctl -p err        # errors and above
journalctl -p warning..err

# Filter by time range
journalctl --since "2026-03-13 08:00" --until "2026-03-13 09:00"
journalctl --since "1 hour ago"

# Follow in real-time
journalctl -f

# Show specific service
journalctl -u nginx.service -b

# Show structured fields
journalctl -o json-pretty -n 5

# Disk usage
journalctl --disk-usage

# Vacuum old logs
journalctl --vacuum-size=100M
journalctl --vacuum-time=7d

journald Configuration (/etc/systemd/journald.conf)

[Journal]
# Persistent storage (requires /var/log/journal/ to exist)
Storage=persistent

# Maximum disk usage by journal files
SystemMaxUse=50M

# Maximum per-boot journal size
SystemMaxFileSize=10M

# Forward to syslog socket (for legacy syslog daemons)
ForwardToSyslog=no

# Forward to kernel log buffer
ForwardToKMsg=no

# Rate limiting — drop messages if a service logs too fast
RateLimitIntervalSec=30s
RateLimitBurst=1000

# Compress entries larger than this
Compress=yes

Structured Logging from Applications

# Shell — use systemd-cat
echo "Deployment complete" | systemd-cat -t myapp -p notice

# With custom fields (key=value, uppercase keys are journal fields)
systemd-cat -t myapp << 'EOF'
BUILD_ID=a3f9b
DEPLOY_ENV=production
Application started successfully
EOF

/* C — use sd-journal API */
#include <systemd/sd-journal.h>

sd_journal_send("MESSAGE=Sensor reading: %.2f°C", temp,
                "PRIORITY=%i", LOG_INFO,
                "SENSOR_ID=0x%02x", sensor_id,
                NULL);

Interview Q&A

{:.gc-iq}

Interview Q&A

Q1 — What is the difference between Type=forking and Type=simple?

Type=simple (the default) tells systemd that the process started by ExecStart is the main service process. systemd marks the service active immediately. Type=forking is for traditional daemons that double-fork — the parent process exits and a child becomes the daemon. systemd tracks the daemon via a PID file (PIDFile=). Using Type=simple for a forking daemon causes problems because systemd would consider the parent’s exit as service failure.

Q2 — What is the difference between WantedBy= and RequiredBy= in [Install]?

WantedBy=multi-user.target means systemctl enable creates a symlink in multi-user.target.wants/. If this service fails to start, the target continues. RequiredBy=multi-user.target creates a symlink in multi-user.target.requires/ — if this service fails, the target itself fails. Use WantedBy= for most services; RequiredBy= only when a target truly cannot function without the service.

Q3 — How does socket activation work, and why is it useful?

systemd opens the socket and holds the file descriptor before the service starts. When a connection arrives, systemd starts the service and passes the socket fd via the SD_LISTEN_FDS_START mechanism. Benefits: parallel boot (all sockets available from the start regardless of service startup order), lazy start (service only starts when first needed), and zero-downtime restart (connections queue in the kernel buffer while the service restarts).

Q4 — What is the difference between After=, Requires=, and Wants=?

After=B means this unit starts only after B is active — it is purely about ordering. Requires=B is a hard dependency — if B fails, this unit fails too, but it does not imply ordering. Wants=B is a soft dependency — B is started alongside this unit, but B’s failure is tolerated. In practice, to express “start after B, and B must succeed”, use both After=B and Requires=B.

Q5 — How do you debug a service that fails to start?

# 1. Check status and last log lines
systemctl status myapp.service

# 2. Check full journal output
journalctl -u myapp.service -b --no-pager

# 3. Check dependency chain
systemctl list-dependencies myapp.service

# 4. Run the ExecStart command manually as the service user
sudo -u myapp /opt/myapp/bin/myapp --config /etc/myapp/config.toml

# 5. Enable debug output
systemctl set-environment SYSTEMD_LOG_LEVEL=debug
systemctl daemon-reload && systemctl restart myapp.service

Q6 — Where should you place custom unit files, and why not in /lib/systemd/system/?

Custom and site-local unit files belong in /etc/systemd/system/. Files there have higher precedence than /lib/systemd/system/ and survive package upgrades that might overwrite /lib/systemd/system/ files. Use systemctl edit servicename to create a drop-in override file at /etc/systemd/system/servicename.d/override.conf, which merges with the original rather than replacing it entirely.

Q7 — How do you make a service restart automatically and limit restart storms?

[Service]
Restart=on-failure
RestartSec=5s
# Allow at most 3 restarts in 60 seconds before giving up
StartLimitIntervalSec=60s
StartLimitBurst=3
# Action when burst limit is hit (none, reboot, poweroff, etc.)
StartLimitAction=none

After the burst limit is exceeded, the service enters the failed state and will not restart until manually reset with systemctl reset-failed myapp.service.

References

{:.gc-ref}

References

Resource Link
systemd Official Documentation systemd.io
man 5 systemd.unit Unit file format reference
man 5 systemd.service Service unit reference
man 5 systemd.socket Socket unit reference
man 5 systemd.timer Timer unit reference
man 1 journalctl Journal query tool
man 5 journald.conf journald configuration
Freedesktop systemd Index freedesktop.org/software/systemd/man/
sd_notify(3) freedesktop.org/software/systemd/man/sd_notify.html
Bootlin systemd for Embedded bootlin.com/training/embedded-linux
Lennart Poettering’s Blog 0pointer.net/blog
Previous BusyBox Root Filesystem Next SysVinit & Runlevels