Module 2 · Lesson 16 of 30

Daemons and systemd services (Type=notify, capabilities)

Build reliable Linux daemons with logging, reloads, systemd readiness, watchdogs, capability limits, sandboxing and socket activation.

  • Beginner to advanced
  • 32 min read
  • 60 min lab
What you will need

Ubuntu 24.04 VM with systemd and administrator access; C compiler, pkg-config, libsystemd-dev and libcap2-bin; optional ARM board

By the end you can 7 objectives
  • Define a daemon and describe its startup, reload and shutdown lifecycle.
  • Explain every step of classic daemon detachment and the limitations of daemon().
  • Implement safe logging, SIGHUP reload handling and locked PID tracking.
  • Create a Type=notify unit with readiness reporting and a meaningful watchdog.
  • Analyze UID changes and capability transformations across execve().
  • Evaluate sandbox, resource and activation settings for an embedded service.
  • Diagnose service failures using unit properties, the journal and process credentials.
Read the lesson below; the video will appear here when published.
01

A service that outlives your login

3 min read

Your board collects temperatures into shared memory from the previous lesson. You launch its collector over SSH, a remote terminal connection. Collection and fault reporting must continue after disconnection.

A Many daemons run for the whole boot. Others start when work arrives and stop when idle. Running as root is not part of the definition.

Appending & to a command starts a background shell job. It does not remove inherited terminal descriptors or create a service restart policy. Terminal hangup and shell job handling can still affect it. nohup changes hangup handling and redirects some terminal output; it does not supervise health or arrange startup after reboot.

A Here, systemd's system instance runs as process ID 1. A process ID, abbreviated PID, identifies a running process within its PID namespace.

Separate these lifecycle events:

Event What your collector must do
Start Read configuration, allocate resources and establish communication
Become ready Promise that its documented work can now be served
Run Process requests, report faults and release temporary resources
Reload Validate replacement settings without corrupting ongoing work
Stop Refuse new work, finish bounded cleanup and exit
Fail Leave useful evidence and let supervision decide about recovery

Successful process creation proves little. A collector may exist while configuration parsing fails or a device remains unavailable. Conversely, a process can remain alive after its worker thread becomes stuck. Readiness and continuing health therefore need separate evidence.

Distinguish required resources from recoverable absences. Missing configuration might prevent startup; a disconnected optional sensor might permit degraded operation. Define this before choosing restart settings.

Inspect the manager before using the service commands. ps selects PID 1 and prints its command name:

bash
ps -p 1 -o comm=

If the result identifies systemd, continue. Containers and BusyBox images may use another init program. WSL needs systemd enabled for these exercises. Classic daemon code does not require systemd.

Service lifecycleInitializingReadyReloadingStoppingStoppedFailedTransitionsInitialization succeedsInitializingReadyReload requestedReadyReloadingAccept or rejectReloadingReadyStop requestedReadyStoppingCleanup completesStoppingStoppedInitialization failsInitializingFailedHealth failureReadyFailedRestart policy permitsFailedInitializingInitializingReadyReloadingStoppingStoppedFailedTransitionsInitialization succeedsInitializingReadyReload requestedReadyReloadingAccept or rejectReloadingReadyStop requestedReadyStoppingCleanup completesStoppingStoppedInitialization failsInitializingFailedHealth failureReadyFailedRestart policy permitsFailedInitializing
A service moves through initialization, readiness, reload and bounded shutdown.
02

Detaching a classic daemon

4 min read

A A session groups related process groups. Its leader's PID equals the session ID. These job-control relationships explain the two forks.

1Fork before creating a session

fork() creates a child process. Check for failure; the parent normally exits with _exit() while the child continues. _exit() avoids running inherited exit handlers or flushing copied C output buffers. The child inherits its process group ID but receives a different PID, so it is not that group's leader.

2Create a session, then fork again

The child calls setsid(). This creates a new session and process group, with no controlling terminal. The call fails if its caller is already a process group leader; the first fork avoids that condition.

The new session leader forks again. Its parent exits, and the final child continues as a nonleader. On Linux, this prevents automatic acquisition of a controlling terminal when it opens a terminal device. Keep O_NOCTTY on later terminal opens too. A deliberate future session change could undo these assumptions; detachment is not a security boundary.

The orphan is adopted by an appropriate child subreaper or PID 1. Do not require its parent PID to equal 1 in a correctness test.

3Set predictable directory and permissions

Check chdir("/") succeeds, so the working directory stops keeping an unrelated filesystem busy. Use absolute paths afterward.

The Classic sequences include umask(0) to remove the launcher's unknown mask. This makes the modes supplied by your program especially important. Request 0600 for private files, or deliberately establish a restrictive mask such as 027 before creating service data. A zero mask does not make unsafe creation modes acceptable.

4Close inherited descriptors and restore standard streams

A file descriptor is a small integer referring to an open kernel object. Close unwanted inherited descriptors, including terminal descriptors. Preserve only explicitly required descriptors, such as a startup-error pipe. On Linux, close_range() can close a range efficiently when supported. Older systems need a carefully bounded fallback or descriptor enumeration; never assume only descriptors 0 through 1023 exist.

Open /dev/null read-write, then use dup2() to install it on descriptors 0, 1 and 2. They represent standard input, output and error. Reads then return end-of-file, and writes are discarded. Leaving them closed risks later opens reusing these numbers, allowing library output to corrupt an unrelated file.

Perform this work before creating threads. In a multithreaded process, the child after fork() has severe restrictions until it executes another program. Locks held by vanished threads are a common deadlock cause.

Step by stepClassic detachment sequence
  1. First fork: The parent exits and the child is not a process group leader.

  2. New session: setsid removes the controlling terminal association.

  3. Second fork: The final process is not a session leader.

  4. Directory and mask: Choose the working directory and file creation policy.

  5. Descriptor cleanup: Close inherited resources except an explicit preserve list.

  6. Standard streams: Map standard input, output and error to /dev/null.

  7. Initialize service: Acquire the instance lock, load settings and report startup success.

The nonstandard libc helper daemon(nochdir, noclose) performs some detachment work. In glibc, daemon(0, 0) changes directory and redirects standard streams, but does not double-fork. It leaves a session leader and does not perform your full descriptor cleanup, locking or umask policy. Treat musl or another libc as a separate implementation to inspect. See daemon(3).

Going deeperA successful launcher exit can hide failed startup

If the original parent exits before the final child initializes, its successful exit status proves only that launching began. Use a pipe created before forking when a classic launcher must report real initialization errors. Preserve the child's write end during descriptor cleanup. Have the parent wait for an explicit success or error record, with a timeout. Only the final daemon owns the long-lived instance lock.

Classic daemon detachmentFirst forkOriginalparentexitssetsidNew sessionwithoutcontrollingterminalSecond forkSurvivor isnot sessionleaderDirectoryand maskChoose rootdirectoryandcreationpolicyClosedescriptorsPreserveonlyexplicitlyrequiredhandlesRestorestandardstreamsMapdescriptorsto/dev/nullInitializeserviceLock,configureand reportstartupFirst forkOriginalparentexitssetsidNew sessionwithoutcontrollingterminalSecond forkSurvivor isnot sessionleaderDirectoryand maskChoose rootdirectoryandcreationpolicyClosedescriptorsPreserveonlyexplicitlyrequiredhandlesRestorestandardstreamsMapdescriptorsto/dev/nullInitializeserviceLock,configureand reportstartup
Two forks separate the final daemon from session leadership and the terminal.
03

Logging, reloads and reliable instance tracking

2 min read

Log the failed operation, resource and next action. Avoid logging every sample: repeated messages consume CPU, storage and flash endurance.

The C syslog() interface submits messages to a local logging service, commonly through /dev/log. A facility identifies the broad source category, such as LOG_DAEMON. A priority gives severity, such as LOG_INFO or LOG_ERR. This userspace API is different from the Linux kernel's syslog(2) interface.

This complete small program sends one informational message. LOG_PID attaches the PID, and LOG_NDELAY requests opening the connection immediately:

c
#include <syslog.h>

int main(void)
{
    openlog("telemetryd", LOG_PID | LOG_NDELAY, LOG_DAEMON);
    syslog(LOG_INFO, "configuration accepted; sampling enabled");
    closelog();
    return 0;
}

closelog() releases the library's logging connection. It does not stop the system logger. Never pass external text as the format string: use syslog(LOG_INFO, "%s", text). syslog() returns no delivery acknowledgment. Its return cannot prove that a message reached persistent storage. See syslog(3).

The systemd journal stores messages with metadata, including service identity. Native logging uses <systemd/sd-journal.h> and libsystemd:

c
#include <syslog.h>
#include <systemd/sd-journal.h>

int main(void)
{
    int rc = sd_journal_print(LOG_NOTICE,
                             "telemetry configuration accepted");
    return rc < 0 ? 1 : 0;
}

sd_journal_print() supplies a formatted message and priority; sd_journal_send() supports additional structured fields. A successful call still does not promise persistence across power loss. Services may also log to standard output and error when the unit connects those streams to the journal. See native journal logging.

To compile these examples, save each separately as syslog-example.c and journal-example.c. The second command gets compiler and linker flags from the installed development package:

bash
cc -std=c11 -Wall -Wextra syslog-example.c -o syslog-example
cc -std=c11 -Wall -Wextra journal-example.c -o journal-example $(pkg-config --cflags --libs libsystemd)

1Receive a reload request safely

SIGHUP originally reports terminal hangup. Daemons often use it to request reconfiguration, but its default action remains termination. Install handling deliberately. Do not parse configuration, call malloc() or call syslog() inside a signal handler: these operations are not async-signal-safe, meaning they are not guaranteed safe during asynchronous interruption.

A handler may set a volatile sig_atomic_t flag. Avoid the test-then-sleep race using blocked signals and pselect(). For threaded services, block control signals before creating workers. Receive them synchronously using sigwaitinfo() in one thread, or Linux signalfd() in a descriptor-based event loop.

2Replace settings only after validation

Parse into a separate candidate configuration. Check ranges, paths and cross-field rules. Acquire any replacement resources before swapping the active configuration under appropriate synchronization. If parsing fails, retain the old settings and log the rejection. Standard signals can merge while pending, so one SIGHUP means “reread the latest file,” not “apply exactly one queued edit.”

3Hold a lock, then record the PID

A PID file is a text file containing the current PID. Its existence cannot prove that its original process still runs. Crashes leave stale files, and PIDs are reused. Checking kill(pid, 0) checks existence or permissions, not executable identity.

For a classic daemon, create a file inside a trusted runtime directory, then acquire flock(fd, LOCK_EX | LOCK_NB). Only after locking should you truncate and write the PID. Keep the descriptor open for the daemon's lifetime. A crash closes its descriptors and releases the lock, though the text file may remain. Handle interrupted and short writes just as in earlier file I/O lessons.

Never unlink a lock file while another process could use its name. Its replacement is a different inode and can acquire a separate lock. For systemd-native services, let the manager track the foreground PID and its processes instead of introducing a PID file.

Reload versus instance trackingConfigurationreloadInstancetrackingInputSIGHUPrequestStart requestFirst safeguardConsumesignal safelyAcquireexclusivelockNext actionValidatecandidateRecord finalPIDKeep valid stateRetain old onfailureHolddescriptoropenCompletionSwap acceptedsettingsExit releaseslock
Reload validation and a held file lock solve different lifecycle problems.
04

Describing a service with a unit

2 min read

A A service unit name ends in .service. Local administrator units belong under /etc/systemd/system/; packaged units belong in the distribution's vendor unit directory.

This unit assumes an installed executable and dedicated telemetry account. You implement its application option --foreground in the lab.

ini
[Unit]
Description=Telemetry collector
After=local-fs.target
StartLimitIntervalSec=60s
StartLimitBurst=5

[Service]
Type=simple
User=telemetry
Group=telemetry
EnvironmentFile=-/etc/telemetryd.env
ExecStart=/usr/local/sbin/telemetryd --foreground
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=3s
TimeoutStopSec=15s
RuntimeDirectory=telemetryd
RuntimeDirectoryMode=0750
StateDirectory=telemetryd
UMask=0027
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

[Unit] describes identity, ordering and dependencies. After= sets ordering when both units start; it does not pull the other unit in. Wants= requests another unit, while Requires= adds stronger failure relationships. A target groups units to represent a startup goal. multi-user.target is a common non-graphical system goal, not a shell login.

[Service] describes the process. User= and Group= select its identity before execution. RuntimeDirectory= creates /run/telemetryd for temporary data; StateDirectory= manages persistent data under /var/lib/telemetryd. Keep executables and configuration directories unwritable by this account.

EnvironmentFile= reads assignment lines such as SAMPLE_INTERVAL=5. Its leading minus permits a missing file. This file is not a shell script: do not expect shell command substitution. Environment variables are unsuitable for secrets because they can spread through child processes and diagnostic interfaces.

ExecStart= does not implicitly invoke a shell. Pipes, redirection and shell built-ins therefore do not work as shell syntax there. $MAINPID in ExecReload= is systemd's expansion for the tracked main process. This reload command sends a request and returns; its completion does not prove that configuration validation finished.

Restart=on-failure handles selected abnormal exits, signals, timeouts and watchdog failures. A clean exit usually does not trigger it. An explicit manager stop does not cause an unwanted restart. Rate limits and RestartSec= prevent a failing service from spinning continuously.

After installing telemetryd.service, verify it and reload manager definitions. Enabling arranges future activation; starting runs it now:

bash
sudo systemd-analyze verify /etc/systemd/system/telemetryd.service
sudo systemctl daemon-reload
sudo systemctl enable telemetryd.service
sudo systemctl start telemetryd.service
systemctl status telemetryd.service
sudo journalctl -u telemetryd.service -b

status combines state and recent messages. journalctl -b restricts messages to this boot. Unit installation, enablement, current activity and application readiness are separate facts. Consult the service unit documentation for their exact relationships.

Service unit sectionstelemetryd.serviceUnitDescription, ordering and dependenciesServiceExecutionType, ExecStart, User, EnvironmentFileRecoveryRestart, timeouts and watchdogStorageRuntimeDirectory and StateDirectoryInstallWantedBy links created during enablementtelemetryd.serviceUnitDescription, ordering anddependenciesServiceExecutionType, ExecStart, User,EnvironmentFileRecoveryRestart, timeouts andwatchdogStorageRuntimeDirectory andStateDirectoryInstallWantedBy links createdduring enablement
Unit sections connect dependencies, process settings and boot-time enablement.
05

Readiness, service types and watchdogs

4 min read

A collector needs time to validate storage. Starting its publisher immediately could lose records. Readiness signaling avoids a guessed startup delay.

Type What marks startup success Suitable use
simple Manager has forked the service process Foreground service without readiness signaling
exec Service executable has successfully executed Foreground service needing execution errors caught
forking Initial parent exits successfully Existing self-backgrounding daemon
oneshot Startup command finishes successfully Bounded setup task
notify Accepted READY=1 notification arrives Foreground service with real initialization checks
dbus Configured bus name is acquired Service whose availability is represented on D-Bus

For forking, set PIDFile= when needed to identify the surviving main process. systemd reads that file; it does not write it for the daemon. For oneshot, RemainAfterExit=yes can retain an active state after commands exit, but it creates no running worker. simple can appear started even before an execution error is discovered. exec fixes that distinction but does not prove application initialization.

For a notify implementation, replace Type=simple with these settings in the previous unit:

ini
Type=notify
NotifyAccess=main
TimeoutStartSec=30s
WatchdogSec=20s

A This service watchdog is separate from a hardware watchdog that can reset the whole board.

1Initialize, then send readiness

Include <systemd/sd-daemon.h> and link libsystemd. Call the following only after your configuration and required resources work. These are statements inside your service's normal startup path:

c
int rc = sd_notify(0, "READY=1\nSTATUS=Collecting telemetry");
if (rc < 0) {
    syslog(LOG_ERR, "readiness notification failed: %s", strerror(-rc));
    return EXIT_FAILURE;
}

The surrounding source also needs <syslog.h>, <string.h> and <stdlib.h>. sd_notify() returns a negative error number on failure, zero when no notification socket is configured, and a positive value when sent. A zero result can be normal during direct foreground testing. The 0 first argument preserves the notification environment for later status and watchdog calls.

2Schedule reports from real progress

sd_watchdog_enabled(0, &interval_usec) reads the watchdog environment and checks whether it applies to the calling process. Use a uint64_t interval from <stdint.h>. A positive result means enabled; zero means disabled; a negative result is an error. The interval is in microseconds. Schedule checks before expiry, commonly at half the interval, using a monotonic clock unaffected by wall-clock corrections. See sd_watchdog_enabled(3).

After confirming that workers and their deadlines are healthy, send sd_notify(0, "WATCHDOG=1"). Send human-readable changes through sd_notify(0, "STATUS=Waiting for sensor reconnect"). STATUS= does not imply readiness or satisfy the watchdog.

Never report health from an independent timer while collection is deadlocked. Check completed work or bounded idleness. For multiple workers, inspect progress counters and completion times against promised deadlines.

3Handle failure and stop coherently

After readiness, watchdog expiry normally terminates the service with SIGABRT; restart policy can then recover it. Missing startup readiness instead reaches TimeoutStartSec=. During orderly shutdown, send STOPPING=1, stop accepting work, complete bounded cleanup and exit. The manager can eventually use SIGKILL when stopping exceeds its configured timeout.

Step by stepReadiness and continuing health
  1. Launch: The manager starts the foreground program with notification settings.

  2. Initialize: The program validates configuration and opens required resources.

  3. Declare ready: The main process sends READY=1 and useful STATUS text.

  4. Release dependents: Ordered follow-up services can proceed after startup completes.

  5. Check progress: The event loop verifies useful work or healthy bounded idleness.

  6. Report health: WATCHDOG=1 arrives before the configured interval expires.

NotifyAccess=main accepts notifications from the main process. A short-lived helper can introduce attribution races, even with broader access. Keep reporting in the tracked service process. Advanced integrations can use sd_notify_barrier() when synchronization is needed. See the notification protocol.

Readiness and health notificationsManagerCollectorResourcesLaunch foregroundprocessValidate andinitializeRequired resourcesavailableREADY=1 and STATUSWork or idle checksProgress remainshealthyWATCHDOG=11. Manager → CollectorLaunch foreground process2. Collector → ResourcesValidate and initialize3. Resources → CollectorRequired resources available4. Collector → ManagerREADY=1 and STATUS5. Collector → ResourcesWork or idle checks6. Resources → CollectorProgress remains healthy7. Collector → ManagerWATCHDOG=1
Initialization, READY and repeated health messages have different meanings.
06

Credentials and permanent privilege removal

3 min read

An older collector starts as root to open a protected device. Drop its authority before handling untrusted requests.

Linux records a real user ID, effective user ID and saved set-user-ID. A UID is a numeric user identifier; root normally has UID zero.

The real UID generally records the originating identity. The effective UID normally determines permissions, with Linux filesystem checks using a related filesystem UID. The saved UID can permit later restoration of an earlier effective identity. Group IDs have corresponding roles. Supplementary groups provide additional access, such as membership in a hardware-access group.

seteuid(uid) changes the effective UID while leaving the saved UID available. It suits temporary changes, not an assumed permanent drop. setreuid(real, effective) changes two IDs; -1 leaves an argument unchanged. Its saved-ID side effects depend on the requested changes. Avoid treating these interfaces as interchangeable. See setreuid(2).

A With CAP_SETUID, setuid(nonroot_uid) replaces real, effective and saved UIDs. Unprivileged callers have narrower rules. setgid() similarly changes group IDs with CAP_SETGID. Check every result. See setuid(2).

For explicit Linux-only permanent changes, use setresgid() and setresuid() to name all three values. The following function assumes a single-threaded, privileged startup phase and already validated, nonzero target IDs. Compile with _GNU_SOURCE defined before headers:

c
#define _GNU_SOURCE
#include <sys/types.h>
#include <stddef.h>
#include <grp.h>
#include <unistd.h>

static int drop_identity(uid_t uid, gid_t gid)
{
    if (setgroups(0, NULL) == -1)
        return -1;
    if (setresgid(gid, gid, gid) == -1)
        return -1;
    if (setresuid(uid, uid, uid) == -1)
        return -1;
    return 0;
}

This selects no supplementary groups. If approved access requires group memberships, call initgroups("telemetry", gid) instead of clearing them. That function consults the group database. Never preserve the launcher's groups by accident. Set groups before dropping the authority needed to change them. Treat any partial failure as fatal; do not serve requests. See initgroups(3) and setresuid(2).

Verify all ID fields with getresuid() and getresgid(), plus getgroups(). Also inspect capability sets. UID changes usually remove capabilities under normal rules, but securebits and capability-retention settings can change that behavior. Securebits are Linux flags controlling special privilege rules. Do not claim complete privilege removal from UID numbers alone.

Going deeperA dropped identity does not revoke open handles

An already opened device or file can remain usable after a UID change. Review every retained descriptor as authority. Linux kernel credentials are per-thread, represented through struct cred; libc wrappers coordinate ID changes between threads to support process-wide POSIX behavior. Raw system calls can bypass that coordination. Drop privileges before starting workers, and avoid direct syscall shortcuts for credential changes.

Prefer User=telemetry when the application needs no privileged startup at all. The manager can establish credentials before your parser, libraries and application initialization run.

Permanent identity changeValidatetarget IDsChoosededicatednonzeroserviceidentitySet supplementarygroupsClear orinitializeapprovedmembershipsReplacegroup IDsReal,effectiveand savedReplaceuser IDsReal,effectiveand savedExit from step 4: Any change failsTerminate startupVerifyremainingauthorityInspectIDs, capabilities anddescriptorsAcceptrequestsOnly afterevery checksucceedsValidatetarget IDsChoosededicatednonzeroserviceidentitySet supplementarygroupsClear orinitializeapprovedmembershipsReplacegroup IDsReal,effectiveand savedReplaceuser IDsReal,effectiveand savedExit from step 4: Any changefailsTerminate startupVerifyremainingauthorityInspectIDs, capabilities anddescriptorsAcceptrequestsOnly afterevery checksucceeds
Remove supplementary access and replace every saved identity before handling requests.
07

Capability sets and execution

4 min read

CAP_NET_BIND_SERVICE permits binding privileged Internet ports. CAP_CHOWN permits otherwise restricted ownership changes. CAP_DAC_OVERRIDE bypasses ordinary discretionary file access checks. Executing a regular file still requires at least one execute permission bit. It does not bypass every security mechanism. CAP_SYS_ADMIN covers many unrelated administration operations, including mounting filesystems and certain namespace operations. It is far too broad for a simple telemetry collector.

Capabilities are per-thread attributes. Tools often call them process capabilities because ordinary applications keep consistent credentials across threads. Five sets matter:

Set Meaning
Effective Powers currently checked for privileged operations
Permitted Powers the thread may place into its effective set
Inheritable Powers eligible to combine with a program file's inheritable set
Ambient Powers carried across ordinary, nonprivileged program execution
Bounding Ceiling applied to file-permitted capability gains during execution

An executable can also carry file capabilities in security.capability. This is an extended attribute, or metadata attached to the file outside its normal contents. It records permitted and inheritable sets plus an effective flag. The file effective value is a flag, not an independent bitmap matching every process capability.

execve() replaces the running program image. In the ordinary non-root case, capability calculation can be summarized as follows. P means the calling thread, F means the executable file, and new means after execution. & means intersection and | means union:

text
new ambient   = privileged file ? empty : old ambient
new permitted = (old inheritable & file inheritable)
              | (file permitted & old bounding)
              | new ambient
new effective = file effective flag ? new permitted : new ambient
new inheritable = old inheritable
new bounding    = old bounding

Here, a privileged file includes a set-user-ID or set-group-ID executable, or one carrying file capabilities. Ambient membership requires membership in both permitted and inheritable sets. Ordinary execution does not preserve arbitrary old permitted capabilities. Also, bounding-set removal alone does not erase a capability already held elsewhere. Root-specific rules, securebits, nosuid, tracing and no_new_privs add important conditions. Use capabilities(7) when analyzing an actual execution path.

1Give a compiled helper one file capability

Assume your lab has produced a native executable bind-helper that binds TCP port 80. A port identifies a network endpoint; binding reserves the local address and port. Build it as a foreground program without a “must be root” check. Install a controlled copy, grant one capability and inspect its metadata:

bash
sudo install -d -o root -g root -m 0755 /usr/local/libexec
sudo install -o root -g root -m 0755 ./bind-helper /usr/local/libexec/bind-helper
sudo setcap 'cap_net_bind_service=ep' /usr/local/libexec/bind-helper
getcap /usr/local/libexec/bind-helper
/usr/local/libexec/bind-helper

Run the final command as the ordinary lab user. Here p selects the file permitted set and e sets its effective flag. getcap reports executable metadata; inspect the running task too. Never grant this capability to a general interpreter: arbitrary scripts would share that power.

2Check the actual privilege boundary

The traditional privileged-port boundary is 1024. The network-namespace setting net.ipv4.ip_unprivileged_port_start can lower it, including to zero. Read it with sysctl net.ipv4.ip_unprivileged_port_start before interpreting success without capabilities. A network namespace supplies a separate networking environment; containers may have different settings.

File capabilities require compatible filesystem metadata. Copying, modifying or replacing the executable can lose or clear them. Verify after deployment. Set-user-ID root usually grants broad root authority to the program; a single file capability limits the initial grant. Powerful capabilities can still be nearly as dangerous. Neither mechanism makes untrusted code safe.

3Prefer a unit-scoped grant where suitable

For a non-root system service that binds its own low port, add:

ini
User=telemetry
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes

The bounding set restricts available powers; it does not grant one by itself. The ambient setting arranges the grant across execution without changing executable metadata. Use an ordinary executable without file capabilities for this variant. NoNewPrivileges=yes blocks new execution-based privilege gains, including file-capability gains, but does not remove capabilities already established by the manager. See the kernel explanation.

Capability sets across execveSetExecutionroleEffectiveUsed forchecksRecalculatedfor new imagePermittedAvailableeffectivepowersFile andinheritedcontributionsInheritableEligibleinheritedpowersIntersectsfileinheritablesetAmbientOrdinaryexecutionbridgeCleared byprivilegedexecutionBoundingAcquisitionceilingMasks filepermittedcontribution
File metadata and thread capability sets determine authority after program execution.
08

Sandboxing and resource limits

3 min read

List the collector's required write paths, devices, network families and helpers. A Restrict access around those needs.

Add these settings under [Service] for a collector that writes only managed state and runtime directories:

ini
PrivateTmp=yes
ProtectSystem=strict
ReadOnlyPaths=/srv/telemetry-calibration
NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=
CPUQuota=25%
MemoryHigh=96M
MemoryMax=128M
TasksMax=32
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native

The calibration directory must exist for this example. PrivateTmp=yes provides private /tmp and /var/tmp views. A filename under host /tmp is therefore unsuitable for communicating with another service. ProtectSystem=strict makes most of the filesystem read-only in the service's mount namespace, a separate view of mounts. Managed StateDirectory= and RuntimeDirectory= locations receive writable exceptions.

ReadOnlyPaths= protects named paths, though strict protection already covers this example. It neither hides readable content nor revokes earlier writable descriptors. Keep ordinary permissions restrictive too.

The empty capability assignments remove unnecessary capabilities. This example assumes a collector that needs no low-port bind privilege. If you choose direct privileged binding, replace both lines with the narrow assignments from the preceding section.

NoNewPrivileges=yes persists across descendants and execution. Existing capabilities still matter; it does not ban every privileged operation.

A Children normally inherit the unit's cgroup, so ordinary forking does not escape service accounting. On cgroup v2, CPU bandwidth limits map to cpu.max; memory settings use files including memory.high and memory.max.

CPUQuota=25% caps aggregate CPU time at a quarter of one CPU, not a quarter of every core. Multiple worker threads share that budget. MemoryHigh= encourages reclaim and throttles pressure before MemoryMax= reaches its hard boundary. If memory cannot be reclaimed, the kernel may invoke out-of-memory killing within the constrained group. TasksMax= counts tasks, including threads. See resource controls and the cgroup v2 interfaces.

Going deeperSeccomp filtering uses SystemCallFilter, not seccomp=

Seccomp is a kernel facility for restricting system calls. systemd unit files use SystemCallFilter= for syscall allowlists or denylists; there is no generic seccomp= service directive. @system-service is a maintained systemd syscall group, not a portable list guaranteed to fit every program. SystemCallErrorNumber=EPERM makes blocked calls return a permission error instead of the usual fatal filtering action. Architecture filters reduce alternate syscall-ABI exposure. An ABI defines the binary interface between compiled code and its execution environment. Inspect groups with systemd-analyze syscall-filter, then test startup, reload, normal work and shutdown. A syscall filter cannot inspect arbitrary pathname strings; use filesystem controls for paths. See execution restrictions.

On ARM, a 32-bit helper may need another syscall architecture. Tight CPU budgets can delay watchdog reports. Measure under contention and retain allocation headroom for reloads.

Complementary service restrictionsIdentityUser, Group and supplementary membershipsCapabilitiesNarrow grants and bounding setFilesystem viewPrivateTmp, ProtectSystem and manageddirectoriesExecution privilegeNoNewPrivileges blocks gains during execveSystem callsSystemCallFilter limits available kerneloperationsResource useCPUQuota, MemoryMax and TasksMaxEach control covers a different boundaryIdentityUser, Group and supplementarymembershipsCapabilitiesNarrow grants and boundingsetFilesystem viewPrivateTmp, ProtectSystem andmanaged directoriesExecution privilegeNoNewPrivileges blocks gainsduring execveSystem callsSystemCallFilter limitsavailable kernel operationsResource useCPUQuota, MemoryMax andTasksMaxEach control covers a differentboundary
Identity, capabilities, mounts, syscall filters and cgroups constrain different operations.
09

Socket and D-Bus activation

3 min read

Socket activation moves endpoint creation into the manager. A socket is a kernel communication endpoint. The manager opens and listens on it, then starts a service when traffic needs handling. This can reduce idle memory use and remove the worker's need for binding privileges.

Use this /etc/systemd/system/telemetryd.socket for an application with an explicit socket-activation mode:

ini
[Unit]
Description=Telemetry listener

[Socket]
ListenStream=127.0.0.1:80
Accept=no

[Install]
WantedBy=sockets.target

ListenStream= creates a stream listener, here bound only to loopback. Accept=no passes the listening socket to one matching service, normally telemetryd.service. Its program must consume inherited descriptors rather than bind the address again. Do not combine this mode with a second independent bind to port 80.

Native services call sd_listen_fds() from <systemd/sd-daemon.h>. It validates the activation environment, including LISTEN_PID and LISTEN_FDS. Descriptors begin at SD_LISTEN_FDS_START, whose value is 3. Validate their type, address family and expected count, using helpers such as sd_is_socket_inet(). Never assume descriptor 3 is valid merely because it is open. The next lesson explains the socket operations themselves.

Accept=yes instead accepts each stream connection in the manager and starts an instance of a service template, commonly telemetryd@.service. Each instance receives a connected socket, not the listener. This works for small independent sessions; expensive per-connection initialization or large connection counts can make it unsuitable. StandardInput=socket is an alternative way to expose a socket to compatible programs. See socket unit behavior.

Enable and start the socket when deploying this mode. For a clean first test, stop an already running collector before switching its startup interface:

bash
sudo systemctl daemon-reload
sudo systemctl enable --now telemetryd.socket
systemctl status telemetryd.socket

Keep capabilities empty if the worker needs no other privilege. The system manager owns the low-port bind. Restarting a worker can preserve the listening endpoint, although existing connections survive only if your design preserves them. Stopping just the service may permit later traffic to activate it again; stop the socket too when closing the entry point.

D-Bus is a message bus through which processes exchange structured messages and request named services. A D-Bus name identifies a service on that bus. Type=dbus with BusName=org.example.Telemetry delays startup completion until the process acquires that name. Losing it causes systemd to treat the service as no longer functional.

Demand activation also needs bus configuration. For traditional dbus-daemon system-bus activation, install a service description with Name=, an execution fallback and SystemdService=telemetryd.service, plus bus policy. Name requests then start the mapped unit. Check image integration, especially with dbus-broker. Acquire the name after initialization. See D-Bus activation configuration.

Activation entry pointsAccept=noAccept=yesD-BusTriggerListenertrafficNewconnectionRequested busnameManager roleOwn listenerAcceptconnectionStart mappedunitWorker receivesListeningsocketConnectedsocketBus requestroutingService formSinglematchingserviceTemplateinstanceConfiguredBusNameReadiness basisChosenservice typeChosenservice typeBus nameacquired
Socket traffic and D-Bus requests trigger services through distinct activation paths.
10

Common mistakes and board-level diagnosis

4 min read

“It works in my terminal” means the terminal supplied something the unit does not. That might be a working directory, environment variable, supplementary group, executable search path or permission. Reproduce the unit's inputs before changing its policy.

Symptom Likely cause Diagnose Fix
Notify startup times out No accepted readiness message Check Type, main PID, logs and NotifyAccess Send readiness from the tracked main process after initialization
Service exits immediately It backgrounded under simple or notify Compare process tree with ExecStart Select foreground mode or correctly describe a legacy forking service
Files become unwritable Mount restrictions or ordinary permissions Check paths, ownership and unit sandbox settings Use managed writable directories and correct ownership
Port bind fails Missing capability, occupied port or wrong address Inspect errno, capabilities and ss -ltnp Correct the exact cause; consider socket activation
Low-port test succeeds unexpectedly Namespace has a lower privilege threshold Read ip_unprivileged_port_start there Test against the actual configured boundary
Reload kills the service Default SIGHUP action remains installed Inspect signal setup and termination status Handle or synchronously consume SIGHUP before requesting reload
Service restarts repeatedly Application failure or watchdog starvation Read Result, exit status and recent logs Fix the cause before increasing limits

Inspect the loaded definition, runtime identity and manager result with these read-only commands:

bash
systemctl cat telemetryd.service
systemctl show telemetryd.service -p MainPID -p Result -p ExecMainStatus -p ControlGroup
sudo journalctl -u telemetryd.service -b --no-pager

systemctl cat reveals overrides. Result records the manager outcome; ExecMainStatus identifies an exit status or signal. Logs explain the failing operation.

Read the live process's security fields next. This shell snippet refuses to inspect PID zero when no main process exists:

bash
pid=$(systemctl show telemetryd.service -p MainPID --value)
if [ "$pid" -gt 0 ]; then
    sudo grep -E '^(Uid|Gid|Groups|Cap(Inh|Prm|Eff|Bnd|Amb)|NoNewPrivs|Seccomp):' "/proc/$pid/status"
fi

UID and GID fields appear as real, effective, saved and filesystem IDs. Capability fields are hexadecimal bitmasks. capsh --decode=MASK, from libcap tools, translates one mask into names. Seccomp reports the mode, not a readable list of installed rules. A process can exit during inspection; repeat the observation instead of assuming its PID remains valid forever.

systemd-analyze security telemetryd.service reviews exposure; its score cannot prove application correctness. Use strace for missing files, failed binds or blocked calls. Tracing affects timing and may suppress file-capability gains. Compare with an untraced launch.

Bound logs, descriptors, queues and reconnect delays. Small leaks accumulate over weeks. Test sensor removal, full storage, invalid reloads and slow peripherals. Device operations may outlast the watchdog; design timeouts and cancellation deliberately.

Verify the target image, not just the development PC

BusyBox init does not interpret systemd units. Minimal images may omit journal persistence, libsystemd, libcap utilities, cgroup controllers or seccomp support. Cross-compile with the target SDK and its pkg-config metadata; do not link the host's libraries into an ARM executable. Confirm kernel configuration and installed manager features on the actual board.

Remote tracing can move large diagnostic streams off limited target storage, as the Bootlin debugging material illustrates with LTTng. Keep routine service logs useful enough that full tracing is an escalation, not a requirement for every fault.

Evidence for service failuresStartuptimeoutType andNotifyAccessMainPID andlogsPermissionfailureUIDs andcapabilitiesMounts andownershipBind failureAddress andlistenerPortthresholdWatchdogfailureWorkerprogressCPUthrottlingMemoryfailureUnit memorylimitsLeaks andqueuesCollect evidence before broadening privileges
Match service symptoms to lifecycle, identity, filesystem, network and resource evidence.
11

Try it

1 min read

Open the lab, Build, supervise and restrict a telemetry daemon, and its practice pack. First implement the classic lifecycle with syslog, a held PID-file lock and safe SIGHUP reload handling. Then package a foreground variant with Type=notify, a progress-based watchdog and the required sandbox settings. Finally convert the supplied port-80 root helper to a single file capability, verify both metadata and running credentials, and compare it with socket activation. Keep startup, reload, shutdown and failure evidence for review; the next lesson, Sockets: TCP and UDP, develops the network code behind these service entry points.

Key takeaways

  • A background shell job still depends on its inherited terminal and resources.
  • Classic daemons detach themselves; systemd-native services stay in the foreground.
  • A PID file records identity; an atomic lock provides single-instance exclusion.
  • READY=1 reports completed initialization, while watchdog messages report continuing health.
  • Drop supplementary groups and saved identities when permanently dropping privileges.
  • Capabilities grant individual powers, and their sets change when programs execute.
  • Combine a dedicated user, narrow capabilities, filesystem restrictions and syscall filtering.
  • Socket activation can remove the service's need to bind a privileged port.

Reflect

  1. Explain to a colleague, in two sentences: define a daemon and describe its startup, reload and shutdown lifecycle.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close