A service that outlives your login
3 min readYour board collects temperatures into shared memory from the previous lesson. You launch its collector over SSH, a remote terminal connection. Collection and fault reporting must continue after disconnection.
A A daemon is a background service process that operates independently of an interactive controlling terminal. Many daemons run for the whole boot. Others start when work arrives and stop when idle. Running as root is not part of the definition.
Appending & to a command starts a background shell job. It does not remove inherited terminal descriptors or create a service restart policy. Terminal hangup and shell job handling can still affect it. nohup changes hangup handling and redirects some terminal output; it does not supervise health or arrange startup after reboot.
A A service manager starts services, tracks their processes and applies startup, shutdown and recovery rules. Here, systemd's system instance runs as process ID 1. A process ID, abbreviated PID, identifies a running process within its PID namespace.
Separate these lifecycle events:
| Event | What your collector must do |
|---|---|
| Start | Read configuration, allocate resources and establish communication |
| Become ready | Promise that its documented work can now be served |
| Run | Process requests, report faults and release temporary resources |
| Reload | Validate replacement settings without corrupting ongoing work |
| Stop | Refuse new work, finish bounded cleanup and exit |
| Fail | Leave useful evidence and let supervision decide about recovery |
Successful process creation proves little. A collector may exist while configuration parsing fails or a device remains unavailable. Conversely, a process can remain alive after its worker thread becomes stuck. Readiness and continuing health therefore need separate evidence.
Distinguish required resources from recoverable absences. Missing configuration might prevent startup; a disconnected optional sensor might permit degraded operation. Define this before choosing restart settings.
Inspect the manager before using the service commands. ps selects PID 1 and prints its command name:
ps -p 1 -o comm=
If the result identifies systemd, continue. Containers and BusyBox images may use another init program. WSL needs systemd enabled for these exercises. Classic daemon code does not require systemd.
Detaching a classic daemon
4 min readA A controlling terminal is the terminal associated with a process session for terminal input and job-control signaling. A session groups related process groups. Its leader's PID equals the session ID. These job-control relationships explain the two forks.
1Fork before creating a session
fork() creates a child process. Check for failure; the parent normally exits with _exit() while the child continues. _exit() avoids running inherited exit handlers or flushing copied C output buffers. The child inherits its process group ID but receives a different PID, so it is not that group's leader.
2Create a session, then fork again
The child calls setsid(). This creates a new session and process group, with no controlling terminal. The call fails if its caller is already a process group leader; the first fork avoids that condition.
The new session leader forks again. Its parent exits, and the final child continues as a nonleader. On Linux, this prevents automatic acquisition of a controlling terminal when it opens a terminal device. Keep O_NOCTTY on later terminal opens too. A deliberate future session change could undo these assumptions; detachment is not a security boundary.
The orphan is adopted by an appropriate child subreaper or PID 1. Do not require its parent PID to equal 1 in a correctness test.
3Set predictable directory and permissions
Check chdir("/") succeeds, so the working directory stops keeping an unrelated filesystem busy. Use absolute paths afterward.
The The umask is a process mask that removes permission bits when files or directories are created. Classic sequences include umask(0) to remove the launcher's unknown mask. This makes the modes supplied by your program especially important. Request 0600 for private files, or deliberately establish a restrictive mask such as 027 before creating service data. A zero mask does not make unsafe creation modes acceptable.
4Close inherited descriptors and restore standard streams
A file descriptor is a small integer referring to an open kernel object. Close unwanted inherited descriptors, including terminal descriptors. Preserve only explicitly required descriptors, such as a startup-error pipe. On Linux, close_range() can close a range efficiently when supported. Older systems need a carefully bounded fallback or descriptor enumeration; never assume only descriptors 0 through 1023 exist.
Open /dev/null read-write, then use dup2() to install it on descriptors 0, 1 and 2. They represent standard input, output and error. Reads then return end-of-file, and writes are discarded. Leaving them closed risks later opens reusing these numbers, allowing library output to corrupt an unrelated file.
Perform this work before creating threads. In a multithreaded process, the child after fork() has severe restrictions until it executes another program. Locks held by vanished threads are a common deadlock cause.
First fork: The parent exits and the child is not a process group leader.
New session: setsid removes the controlling terminal association.
Second fork: The final process is not a session leader.
Directory and mask: Choose the working directory and file creation policy.
Descriptor cleanup: Close inherited resources except an explicit preserve list.
Standard streams: Map standard input, output and error to /dev/null.
Initialize service: Acquire the instance lock, load settings and report startup success.
The nonstandard libc helper daemon(nochdir, noclose) performs some detachment work. In glibc, daemon(0, 0) changes directory and redirects standard streams, but does not double-fork. It leaves a session leader and does not perform your full descriptor cleanup, locking or umask policy. Treat musl or another libc as a separate implementation to inspect. See daemon(3).
Going deeperA successful launcher exit can hide failed startup
If the original parent exits before the final child initializes, its successful exit status proves only that launching began. Use a pipe created before forking when a classic launcher must report real initialization errors. Preserve the child's write end during descriptor cleanup. Have the parent wait for an explicit success or error record, with a timeout. Only the final daemon owns the long-lived instance lock.
Logging, reloads and reliable instance tracking
2 min readLog the failed operation, resource and next action. Avoid logging every sample: repeated messages consume CPU, storage and flash endurance.
The C syslog() interface submits messages to a local logging service, commonly through /dev/log. A facility identifies the broad source category, such as LOG_DAEMON. A priority gives severity, such as LOG_INFO or LOG_ERR. This userspace API is different from the Linux kernel's syslog(2) interface.
This complete small program sends one informational message. LOG_PID attaches the PID, and LOG_NDELAY requests opening the connection immediately:
#include <syslog.h>
int main(void)
{
openlog("telemetryd", LOG_PID | LOG_NDELAY, LOG_DAEMON);
syslog(LOG_INFO, "configuration accepted; sampling enabled");
closelog();
return 0;
}
closelog() releases the library's logging connection. It does not stop the system logger. Never pass external text as the format string: use syslog(LOG_INFO, "%s", text). syslog() returns no delivery acknowledgment. Its return cannot prove that a message reached persistent storage. See syslog(3).
The systemd journal stores messages with metadata, including service identity. Native logging uses <systemd/sd-journal.h> and libsystemd:
#include <syslog.h>
#include <systemd/sd-journal.h>
int main(void)
{
int rc = sd_journal_print(LOG_NOTICE,
"telemetry configuration accepted");
return rc < 0 ? 1 : 0;
}
sd_journal_print() supplies a formatted message and priority; sd_journal_send() supports additional structured fields. A successful call still does not promise persistence across power loss. Services may also log to standard output and error when the unit connects those streams to the journal. See native journal logging.
To compile these examples, save each separately as syslog-example.c and journal-example.c. The second command gets compiler and linker flags from the installed development package:
cc -std=c11 -Wall -Wextra syslog-example.c -o syslog-example
cc -std=c11 -Wall -Wextra journal-example.c -o journal-example $(pkg-config --cflags --libs libsystemd)
1Receive a reload request safely
SIGHUP originally reports terminal hangup. Daemons often use it to request reconfiguration, but its default action remains termination. Install handling deliberately. Do not parse configuration, call malloc() or call syslog() inside a signal handler: these operations are not async-signal-safe, meaning they are not guaranteed safe during asynchronous interruption.
A handler may set a volatile sig_atomic_t flag. Avoid the test-then-sleep race using blocked signals and pselect(). For threaded services, block control signals before creating workers. Receive them synchronously using sigwaitinfo() in one thread, or Linux signalfd() in a descriptor-based event loop.
2Replace settings only after validation
Parse into a separate candidate configuration. Check ranges, paths and cross-field rules. Acquire any replacement resources before swapping the active configuration under appropriate synchronization. If parsing fails, retain the old settings and log the rejection. Standard signals can merge while pending, so one SIGHUP means “reread the latest file,” not “apply exactly one queued edit.”
3Hold a lock, then record the PID
A PID file is a text file containing the current PID. Its existence cannot prove that its original process still runs. Crashes leave stale files, and PIDs are reused. Checking kill(pid, 0) checks existence or permissions, not executable identity.
For a classic daemon, create a file inside a trusted runtime directory, then acquire flock(fd, LOCK_EX | LOCK_NB). Only after locking should you truncate and write the PID. Keep the descriptor open for the daemon's lifetime. A crash closes its descriptors and releases the lock, though the text file may remain. Handle interrupted and short writes just as in earlier file I/O lessons.
Never unlink a lock file while another process could use its name. Its replacement is a different inode and can acquire a separate lock. For systemd-native services, let the manager track the foreground PID and its processes instead of introducing a PID file.
Describing a service with a unit
2 min readA A systemd unit is a named object describing a managed resource, such as a service, socket or startup target. A service unit name ends in .service. Local administrator units belong under /etc/systemd/system/; packaged units belong in the distribution's vendor unit directory.
This unit assumes an installed executable and dedicated telemetry account. You implement its application option --foreground in the lab.
[Unit]
Description=Telemetry collector
After=local-fs.target
StartLimitIntervalSec=60s
StartLimitBurst=5
[Service]
Type=simple
User=telemetry
Group=telemetry
EnvironmentFile=-/etc/telemetryd.env
ExecStart=/usr/local/sbin/telemetryd --foreground
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=3s
TimeoutStopSec=15s
RuntimeDirectory=telemetryd
RuntimeDirectoryMode=0750
StateDirectory=telemetryd
UMask=0027
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
[Unit] describes identity, ordering and dependencies. After= sets ordering when both units start; it does not pull the other unit in. Wants= requests another unit, while Requires= adds stronger failure relationships. A target groups units to represent a startup goal. multi-user.target is a common non-graphical system goal, not a shell login.
[Service] describes the process. User= and Group= select its identity before execution. RuntimeDirectory= creates /run/telemetryd for temporary data; StateDirectory= manages persistent data under /var/lib/telemetryd. Keep executables and configuration directories unwritable by this account.
EnvironmentFile= reads assignment lines such as SAMPLE_INTERVAL=5. Its leading minus permits a missing file. This file is not a shell script: do not expect shell command substitution. Environment variables are unsuitable for secrets because they can spread through child processes and diagnostic interfaces.
ExecStart= does not implicitly invoke a shell. Pipes, redirection and shell built-ins therefore do not work as shell syntax there. $MAINPID in ExecReload= is systemd's expansion for the tracked main process. This reload command sends a request and returns; its completion does not prove that configuration validation finished.
Restart=on-failure handles selected abnormal exits, signals, timeouts and watchdog failures. A clean exit usually does not trigger it. An explicit manager stop does not cause an unwanted restart. Rate limits and RestartSec= prevent a failing service from spinning continuously.
After installing telemetryd.service, verify it and reload manager definitions. Enabling arranges future activation; starting runs it now:
sudo systemd-analyze verify /etc/systemd/system/telemetryd.service
sudo systemctl daemon-reload
sudo systemctl enable telemetryd.service
sudo systemctl start telemetryd.service
systemctl status telemetryd.service
sudo journalctl -u telemetryd.service -b
status combines state and recent messages. journalctl -b restricts messages to this boot. Unit installation, enablement, current activity and application readiness are separate facts. Consult the service unit documentation for their exact relationships.
Readiness, service types and watchdogs
4 min readA collector needs time to validate storage. Starting its publisher immediately could lose records. Readiness signaling avoids a guessed startup delay.
| Type | What marks startup success | Suitable use |
|---|---|---|
simple |
Manager has forked the service process | Foreground service without readiness signaling |
exec |
Service executable has successfully executed | Foreground service needing execution errors caught |
forking |
Initial parent exits successfully | Existing self-backgrounding daemon |
oneshot |
Startup command finishes successfully | Bounded setup task |
notify |
Accepted READY=1 notification arrives |
Foreground service with real initialization checks |
dbus |
Configured bus name is acquired | Service whose availability is represented on D-Bus |
For forking, set PIDFile= when needed to identify the surviving main process. systemd reads that file; it does not write it for the daemon. For oneshot, RemainAfterExit=yes can retain an active state after commands exit, but it creates no running worker. simple can appear started even before an execution error is discovered. exec fixes that distinction but does not prove application initialization.
For a notify implementation, replace Type=simple with these settings in the previous unit:
Type=notify
NotifyAccess=main
TimeoutStartSec=30s
WatchdogSec=20s
A A watchdog detects missing health reports within a configured interval and triggers a recovery action. This service watchdog is separate from a hardware watchdog that can reset the whole board.
1Initialize, then send readiness
Include <systemd/sd-daemon.h> and link libsystemd. Call the following only after your configuration and required resources work. These are statements inside your service's normal startup path:
int rc = sd_notify(0, "READY=1\nSTATUS=Collecting telemetry");
if (rc < 0) {
syslog(LOG_ERR, "readiness notification failed: %s", strerror(-rc));
return EXIT_FAILURE;
}
The surrounding source also needs <syslog.h>, <string.h> and <stdlib.h>. sd_notify() returns a negative error number on failure, zero when no notification socket is configured, and a positive value when sent. A zero result can be normal during direct foreground testing. The 0 first argument preserves the notification environment for later status and watchdog calls.
2Schedule reports from real progress
sd_watchdog_enabled(0, &interval_usec) reads the watchdog environment and checks whether it applies to the calling process. Use a uint64_t interval from <stdint.h>. A positive result means enabled; zero means disabled; a negative result is an error. The interval is in microseconds. Schedule checks before expiry, commonly at half the interval, using a monotonic clock unaffected by wall-clock corrections. See sd_watchdog_enabled(3).
After confirming that workers and their deadlines are healthy, send sd_notify(0, "WATCHDOG=1"). Send human-readable changes through sd_notify(0, "STATUS=Waiting for sensor reconnect"). STATUS= does not imply readiness or satisfy the watchdog.
Never report health from an independent timer while collection is deadlocked. Check completed work or bounded idleness. For multiple workers, inspect progress counters and completion times against promised deadlines.
3Handle failure and stop coherently
After readiness, watchdog expiry normally terminates the service with SIGABRT; restart policy can then recover it. Missing startup readiness instead reaches TimeoutStartSec=. During orderly shutdown, send STOPPING=1, stop accepting work, complete bounded cleanup and exit. The manager can eventually use SIGKILL when stopping exceeds its configured timeout.
Launch: The manager starts the foreground program with notification settings.
Initialize: The program validates configuration and opens required resources.
Declare ready: The main process sends READY=1 and useful STATUS text.
Release dependents: Ordered follow-up services can proceed after startup completes.
Check progress: The event loop verifies useful work or healthy bounded idleness.
Report health: WATCHDOG=1 arrives before the configured interval expires.
NotifyAccess=main accepts notifications from the main process. A short-lived helper can introduce attribution races, even with broader access. Keep reporting in the tracked service process. Advanced integrations can use sd_notify_barrier() when synchronization is needed. See the notification protocol.
Credentials and permanent privilege removal
3 min readAn older collector starts as root to open a protected device. Drop its authority before handling untrusted requests.
Credentials are the user IDs, group IDs and other security attributes the kernel uses to decide what a task may do. Linux records a real user ID, effective user ID and saved set-user-ID. A UID is a numeric user identifier; root normally has UID zero.
The real UID generally records the originating identity. The effective UID normally determines permissions, with Linux filesystem checks using a related filesystem UID. The saved UID can permit later restoration of an earlier effective identity. Group IDs have corresponding roles. Supplementary groups provide additional access, such as membership in a hardware-access group.
seteuid(uid) changes the effective UID while leaving the saved UID available. It suits temporary changes, not an assumed permanent drop. setreuid(real, effective) changes two IDs; -1 leaves an argument unchanged. Its saved-ID side effects depend on the requested changes. Avoid treating these interfaces as interchangeable. See setreuid(2).
A A Linux capability is a named privilege allowing specific operations beyond ordinary user and group permissions. With CAP_SETUID, setuid(nonroot_uid) replaces real, effective and saved UIDs. Unprivileged callers have narrower rules. setgid() similarly changes group IDs with CAP_SETGID. Check every result. See setuid(2).
For explicit Linux-only permanent changes, use setresgid() and setresuid() to name all three values. The following function assumes a single-threaded, privileged startup phase and already validated, nonzero target IDs. Compile with _GNU_SOURCE defined before headers:
#define _GNU_SOURCE
#include <sys/types.h>
#include <stddef.h>
#include <grp.h>
#include <unistd.h>
static int drop_identity(uid_t uid, gid_t gid)
{
if (setgroups(0, NULL) == -1)
return -1;
if (setresgid(gid, gid, gid) == -1)
return -1;
if (setresuid(uid, uid, uid) == -1)
return -1;
return 0;
}
This selects no supplementary groups. If approved access requires group memberships, call initgroups("telemetry", gid) instead of clearing them. That function consults the group database. Never preserve the launcher's groups by accident. Set groups before dropping the authority needed to change them. Treat any partial failure as fatal; do not serve requests. See initgroups(3) and setresuid(2).
Verify all ID fields with getresuid() and getresgid(), plus getgroups(). Also inspect capability sets. UID changes usually remove capabilities under normal rules, but securebits and capability-retention settings can change that behavior. Securebits are Linux flags controlling special privilege rules. Do not claim complete privilege removal from UID numbers alone.
Going deeperA dropped identity does not revoke open handles
An already opened device or file can remain usable after a UID change. Review every retained descriptor as authority. Linux kernel credentials are per-thread, represented through struct cred; libc wrappers coordinate ID changes between threads to support process-wide POSIX behavior. Raw system calls can bypass that coordination. Drop privileges before starting workers, and avoid direct syscall shortcuts for credential changes.
Prefer User=telemetry when the application needs no privileged startup at all. The manager can establish credentials before your parser, libraries and application initialization run.
Capability sets and execution
4 min readCAP_NET_BIND_SERVICE permits binding privileged Internet ports. CAP_CHOWN permits otherwise restricted ownership changes. CAP_DAC_OVERRIDE bypasses ordinary discretionary file access checks. Executing a regular file still requires at least one execute permission bit. It does not bypass every security mechanism. CAP_SYS_ADMIN covers many unrelated administration operations, including mounting filesystems and certain namespace operations. It is far too broad for a simple telemetry collector.
Capabilities are per-thread attributes. Tools often call them process capabilities because ordinary applications keep consistent credentials across threads. Five sets matter:
| Set | Meaning |
|---|---|
| Effective | Powers currently checked for privileged operations |
| Permitted | Powers the thread may place into its effective set |
| Inheritable | Powers eligible to combine with a program file's inheritable set |
| Ambient | Powers carried across ordinary, nonprivileged program execution |
| Bounding | Ceiling applied to file-permitted capability gains during execution |
An executable can also carry file capabilities in security.capability. This is an extended attribute, or metadata attached to the file outside its normal contents. It records permitted and inheritable sets plus an effective flag. The file effective value is a flag, not an independent bitmap matching every process capability.
execve() replaces the running program image. In the ordinary non-root case, capability calculation can be summarized as follows. P means the calling thread, F means the executable file, and new means after execution. & means intersection and | means union:
new ambient = privileged file ? empty : old ambient
new permitted = (old inheritable & file inheritable)
| (file permitted & old bounding)
| new ambient
new effective = file effective flag ? new permitted : new ambient
new inheritable = old inheritable
new bounding = old bounding
Here, a privileged file includes a set-user-ID or set-group-ID executable, or one carrying file capabilities. Ambient membership requires membership in both permitted and inheritable sets. Ordinary execution does not preserve arbitrary old permitted capabilities. Also, bounding-set removal alone does not erase a capability already held elsewhere. Root-specific rules, securebits, nosuid, tracing and no_new_privs add important conditions. Use capabilities(7) when analyzing an actual execution path.
1Give a compiled helper one file capability
Assume your lab has produced a native executable bind-helper that binds TCP port 80. A port identifies a network endpoint; binding reserves the local address and port. Build it as a foreground program without a “must be root” check. Install a controlled copy, grant one capability and inspect its metadata:
sudo install -d -o root -g root -m 0755 /usr/local/libexec
sudo install -o root -g root -m 0755 ./bind-helper /usr/local/libexec/bind-helper
sudo setcap 'cap_net_bind_service=ep' /usr/local/libexec/bind-helper
getcap /usr/local/libexec/bind-helper
/usr/local/libexec/bind-helper
Run the final command as the ordinary lab user. Here p selects the file permitted set and e sets its effective flag. getcap reports executable metadata; inspect the running task too. Never grant this capability to a general interpreter: arbitrary scripts would share that power.
2Check the actual privilege boundary
The traditional privileged-port boundary is 1024. The network-namespace setting net.ipv4.ip_unprivileged_port_start can lower it, including to zero. Read it with sysctl net.ipv4.ip_unprivileged_port_start before interpreting success without capabilities. A network namespace supplies a separate networking environment; containers may have different settings.
File capabilities require compatible filesystem metadata. Copying, modifying or replacing the executable can lose or clear them. Verify after deployment. Set-user-ID root usually grants broad root authority to the program; a single file capability limits the initial grant. Powerful capabilities can still be nearly as dangerous. Neither mechanism makes untrusted code safe.
3Prefer a unit-scoped grant where suitable
For a non-root system service that binds its own low port, add:
User=telemetry
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
The bounding set restricts available powers; it does not grant one by itself. The ambient setting arranges the grant across execution without changing executable metadata. Use an ordinary executable without file capabilities for this variant. NoNewPrivileges=yes blocks new execution-based privilege gains, including file-capability gains, but does not remove capabilities already established by the manager. See the kernel explanation.
Sandboxing and resource limits
3 min readList the collector's required write paths, devices, network families and helpers. A A sandbox limits which resources and operations a program can access while it runs. Restrict access around those needs.
Add these settings under [Service] for a collector that writes only managed state and runtime directories:
PrivateTmp=yes
ProtectSystem=strict
ReadOnlyPaths=/srv/telemetry-calibration
NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=
CPUQuota=25%
MemoryHigh=96M
MemoryMax=128M
TasksMax=32
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
The calibration directory must exist for this example. PrivateTmp=yes provides private /tmp and /var/tmp views. A filename under host /tmp is therefore unsuitable for communicating with another service. ProtectSystem=strict makes most of the filesystem read-only in the service's mount namespace, a separate view of mounts. Managed StateDirectory= and RuntimeDirectory= locations receive writable exceptions.
ReadOnlyPaths= protects named paths, though strict protection already covers this example. It neither hides readable content nor revokes earlier writable descriptors. Keep ordinary permissions restrictive too.
The empty capability assignments remove unnecessary capabilities. This example assumes a collector that needs no low-port bind privilege. If you choose direct privileged binding, replace both lines with the narrow assignments from the preceding section.
NoNewPrivileges=yes persists across descendants and execution. Existing capabilities still matter; it does not ban every privileged operation.
A A control group, or cgroup, groups tasks so the kernel can account for and limit their resource use. Children normally inherit the unit's cgroup, so ordinary forking does not escape service accounting. On cgroup v2, CPU bandwidth limits map to cpu.max; memory settings use files including memory.high and memory.max.
CPUQuota=25% caps aggregate CPU time at a quarter of one CPU, not a quarter of every core. Multiple worker threads share that budget. MemoryHigh= encourages reclaim and throttles pressure before MemoryMax= reaches its hard boundary. If memory cannot be reclaimed, the kernel may invoke out-of-memory killing within the constrained group. TasksMax= counts tasks, including threads. See resource controls and the cgroup v2 interfaces.
Going deeperSeccomp filtering uses SystemCallFilter, not seccomp=
Seccomp is a kernel facility for restricting system calls. systemd unit files use SystemCallFilter= for syscall allowlists or denylists; there is no generic seccomp= service directive. @system-service is a maintained systemd syscall group, not a portable list guaranteed to fit every program. SystemCallErrorNumber=EPERM makes blocked calls return a permission error instead of the usual fatal filtering action. Architecture filters reduce alternate syscall-ABI exposure. An ABI defines the binary interface between compiled code and its execution environment. Inspect groups with systemd-analyze syscall-filter, then test startup, reload, normal work and shutdown. A syscall filter cannot inspect arbitrary pathname strings; use filesystem controls for paths. See execution restrictions.
On ARM, a 32-bit helper may need another syscall architecture. Tight CPU budgets can delay watchdog reports. Measure under contention and retain allocation headroom for reloads.
Socket and D-Bus activation
3 min readSocket activation moves endpoint creation into the manager. A socket is a kernel communication endpoint. The manager opens and listens on it, then starts a service when traffic needs handling. This can reduce idle memory use and remove the worker's need for binding privileges.
Use this /etc/systemd/system/telemetryd.socket for an application with an explicit socket-activation mode:
[Unit]
Description=Telemetry listener
[Socket]
ListenStream=127.0.0.1:80
Accept=no
[Install]
WantedBy=sockets.target
ListenStream= creates a stream listener, here bound only to loopback. Accept=no passes the listening socket to one matching service, normally telemetryd.service. Its program must consume inherited descriptors rather than bind the address again. Do not combine this mode with a second independent bind to port 80.
Native services call sd_listen_fds() from <systemd/sd-daemon.h>. It validates the activation environment, including LISTEN_PID and LISTEN_FDS. Descriptors begin at SD_LISTEN_FDS_START, whose value is 3. Validate their type, address family and expected count, using helpers such as sd_is_socket_inet(). Never assume descriptor 3 is valid merely because it is open. The next lesson explains the socket operations themselves.
Accept=yes instead accepts each stream connection in the manager and starts an instance of a service template, commonly telemetryd@.service. Each instance receives a connected socket, not the listener. This works for small independent sessions; expensive per-connection initialization or large connection counts can make it unsuitable. StandardInput=socket is an alternative way to expose a socket to compatible programs. See socket unit behavior.
Enable and start the socket when deploying this mode. For a clean first test, stop an already running collector before switching its startup interface:
sudo systemctl daemon-reload
sudo systemctl enable --now telemetryd.socket
systemctl status telemetryd.socket
Keep capabilities empty if the worker needs no other privilege. The system manager owns the low-port bind. Restarting a worker can preserve the listening endpoint, although existing connections survive only if your design preserves them. Stopping just the service may permit later traffic to activate it again; stop the socket too when closing the entry point.
D-Bus is a message bus through which processes exchange structured messages and request named services. A D-Bus name identifies a service on that bus. Type=dbus with BusName=org.example.Telemetry delays startup completion until the process acquires that name. Losing it causes systemd to treat the service as no longer functional.
Demand activation also needs bus configuration. For traditional dbus-daemon system-bus activation, install a service description with Name=, an execution fallback and SystemdService=telemetryd.service, plus bus policy. Name requests then start the mapped unit. Check image integration, especially with dbus-broker. Acquire the name after initialization. See D-Bus activation configuration.
Common mistakes and board-level diagnosis
4 min read“It works in my terminal” means the terminal supplied something the unit does not. That might be a working directory, environment variable, supplementary group, executable search path or permission. Reproduce the unit's inputs before changing its policy.
| Symptom | Likely cause | Diagnose | Fix |
|---|---|---|---|
| Notify startup times out | No accepted readiness message | Check Type, main PID, logs and NotifyAccess | Send readiness from the tracked main process after initialization |
| Service exits immediately | It backgrounded under simple or notify | Compare process tree with ExecStart | Select foreground mode or correctly describe a legacy forking service |
| Files become unwritable | Mount restrictions or ordinary permissions | Check paths, ownership and unit sandbox settings | Use managed writable directories and correct ownership |
| Port bind fails | Missing capability, occupied port or wrong address | Inspect errno, capabilities and ss -ltnp |
Correct the exact cause; consider socket activation |
| Low-port test succeeds unexpectedly | Namespace has a lower privilege threshold | Read ip_unprivileged_port_start there | Test against the actual configured boundary |
| Reload kills the service | Default SIGHUP action remains installed | Inspect signal setup and termination status | Handle or synchronously consume SIGHUP before requesting reload |
| Service restarts repeatedly | Application failure or watchdog starvation | Read Result, exit status and recent logs | Fix the cause before increasing limits |
Inspect the loaded definition, runtime identity and manager result with these read-only commands:
systemctl cat telemetryd.service
systemctl show telemetryd.service -p MainPID -p Result -p ExecMainStatus -p ControlGroup
sudo journalctl -u telemetryd.service -b --no-pager
systemctl cat reveals overrides. Result records the manager outcome; ExecMainStatus identifies an exit status or signal. Logs explain the failing operation.
Read the live process's security fields next. This shell snippet refuses to inspect PID zero when no main process exists:
pid=$(systemctl show telemetryd.service -p MainPID --value)
if [ "$pid" -gt 0 ]; then
sudo grep -E '^(Uid|Gid|Groups|Cap(Inh|Prm|Eff|Bnd|Amb)|NoNewPrivs|Seccomp):' "/proc/$pid/status"
fi
UID and GID fields appear as real, effective, saved and filesystem IDs. Capability fields are hexadecimal bitmasks. capsh --decode=MASK, from libcap tools, translates one mask into names. Seccomp reports the mode, not a readable list of installed rules. A process can exit during inspection; repeat the observation instead of assuming its PID remains valid forever.
systemd-analyze security telemetryd.service reviews exposure; its score cannot prove application correctness. Use strace for missing files, failed binds or blocked calls. Tracing affects timing and may suppress file-capability gains. Compare with an untraced launch.
Bound logs, descriptors, queues and reconnect delays. Small leaks accumulate over weeks. Test sensor removal, full storage, invalid reloads and slow peripherals. Device operations may outlast the watchdog; design timeouts and cancellation deliberately.
BusyBox init does not interpret systemd units. Minimal images may omit journal persistence, libsystemd, libcap utilities, cgroup controllers or seccomp support. Cross-compile with the target SDK and its pkg-config metadata; do not link the host's libraries into an ARM executable. Confirm kernel configuration and installed manager features on the actual board.
Remote tracing can move large diagnostic streams off limited target storage, as the Bootlin debugging material illustrates with LTTng. Keep routine service logs useful enough that full tracing is an escalation, not a requirement for every fault.
Try it
1 min readOpen the lab, Build, supervise and restrict a telemetry daemon, and its practice pack. First implement the classic lifecycle with syslog, a held PID-file lock and safe SIGHUP reload handling. Then package a foreground variant with Type=notify, a progress-based watchdog and the required sandbox settings. Finally convert the supplied port-80 root helper to a single file capability, verify both metadata and running credentials, and compare it with socket activation. Keep startup, reload, shutdown and failure evidence for review; the next lesson, Sockets: TCP and UDP, develops the network code behind these service entry points.
Key takeaways
- A background shell job still depends on its inherited terminal and resources.
- Classic daemons detach themselves; systemd-native services stay in the foreground.
- A PID file records identity; an atomic lock provides single-instance exclusion.
- READY=1 reports completed initialization, while watchdog messages report continuing health.
- Drop supplementary groups and saved identities when permanently dropping privileges.
- Capabilities grant individual powers, and their sets change when programs execute.
- Combine a dedicated user, narrow capabilities, filesystem restrictions and syscall filtering.
- Socket activation can remove the service's need to bind a privileged port.
Reflect
- Explain to a colleague, in two sentences: define a daemon and describe its startup, reload and shutdown lifecycle.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Official docssystemd.service: service types and lifecycle
- Official docssystemd.exec: credentials and sandboxing
- Official docssd_notify: readiness and status protocol
- Official docssystemd.socket: socket activation
- Official docssystemd.resource-control: cgroup limits
- Official docssd_journal_print: native journal logging
- Man pagecapabilities(7)
- Man pagedaemon(3)
Official documentation first. Links checked when the lesson was published.