Why harden: the attacker's arithmetic
3 min readEvery Linux system ships with choices made for convenience, not for your threat model. Services listen on the network. Accounts exist that nobody uses. Programs run with more power than they need. Hardening is the deliberate work of removing unnecessary access and limiting the damage when something is compromised. It is not one tool or one switch. It is a habit you apply to every layer of the system.
The stakes are concrete. In November 1988 the Morris worm spread across the early internet by exploiting a debug option in sendmail and weak passwords. It infected thousands of machines and showed that a network-facing service is a target the moment it exists. In 2016 the Mirai botnet scanned the internet for embedded devices, mostly cameras and routers, and logged in with factory default usernames and passwords over Telnet and SSH. Hundreds of thousands of boards were enslaved and used to knock large websites offline. The lesson for you is direct: a Raspberry Pi with SSH open and a weak password is found by automated scanners within hours, not weeks.
Three ideas organise everything in this lesson. The The attack surface is the sum of all points where an outsider can try to enter or influence a system: ports, services, accounts and files. is what you shrink first. Least privilege means giving each user and program only the access it needs, and nothing more. is what you apply to everything that remains. Defence in depth means stacking independent protections so that one failure does not expose the whole system. is why you keep going after the obvious fixes: a firewall, then keys-only SSH, then tight sudo rules, then file permissions, then confinement, then logging. An attacker must beat every layer. You only need one layer to hold.
This lesson climbs from a read-only audit to expert tools: capabilities that replace set-user-ID root, and confinement that restricts even root. You own the building blocks already: users and permissions from S03A, SSH and firewalls from S07B, troubleshooting from S10B. Here you point them at one question: what can an attacker reach, and how do you make that smaller?
Embedded boards sharpen every point. A sensor gateway sits unattended in a field cabinet, often on a public address, with little RAM, flash that wears out, and a serial console anyone can plug into. Each section carries the board angle, and the labs run against your Pi.
Map and shrink the attack surface
3 min readStart with the network. A listening socket, a network endpoint where a service accepts incoming connections on a port, is a door. The tool ss lists them. Run it with sudo so it can name the owning process of every socket, not just yours:
sudo ss -tulpn
The flags mean: -t TCP, -u UDP, -l listening only, -p show the process, -n show numbers instead of resolving names. Typical output (yours will differ):
Netid State Local Address:Port Process
tcp LISTEN 0.0.0.0:22 users:(("sshd",pid=812,fd=3))
tcp LISTEN [::]:22 users:(("sshd",pid=812,fd=4))
udp UNCONN 127.0.0.1:323 users:(("chronyd",pid=640,fd=5))
tcp LISTEN *:8080 users:(("python3",pid=2310,fd=3))
The local address column decides who can knock. 0.0.0.0:22 and [::]:22 mean SSH accepts connections on every interface. 127.0.0.1:323 is chrony's command port bound to loopback: only local processes can reach it. *:8080 is a development server exposed to the world. The rule: a loopback bind is local-only; a wildcard bind is exposed. For each exposed listener ask: does this need to be reachable from outside? If not, remove it or bind it to loopback.
1Inventory enabled services
Next list what starts at boot. A service that is enabled but not currently running is still surface, because it will listen after the next reboot:
systemctl list-unit-files --state=enabled
systemctl list-units --type=service --state=running
A full image enables printing (cups), Bluetooth, avahi mDNS discovery and more; on a headless board most are pointless. Disable and stop what you do not need:
sudo systemctl disable --now cups
sudo systemctl disable --now avahi-daemon
disable removes the start-at-boot links; --now also stops the running instance. Verify with ss -tulpn that the port is gone. If it remains, the service may be socket-activated: systemd holds the listener itself, as with ssh.socket in S07B. Check systemctl list-units --type=socket and disable the socket too. systemctl mask is stronger than disable: it links the unit to /dev/null so nothing can start it, even as a dependency.
2Count accounts and close the network edge
Accounts are surface too. Every account with a password, a shell and SSH eligibility is a login an attacker can guess at. You will audit them in the next section. At the network edge, keep the default-deny firewall you built in S07B: allow only the ports you deliberately expose, drop the rest. With ufw the shape is sudo ufw default deny incoming, then explicit allows such as sudo ufw allow 22/tcp. A firewall does not make a weak service safe; it hides the services you forgot about. Both matter.
Raspberry Pi OS enables a few services you may not want on a headless sensor node. Run the same inventory over SSH. Keep ssh, and consider whether you need anything else at all. A board that only reads sensors and uploads data often needs exactly one listening port.
Account security: passwords, aging and audits
5 min readYou met /etc/passwd and /etc/shadow in S03A. Recall the split: passwd holds account metadata and is world-readable; shadow holds password hashes and is readable only by root and the shadow group. On Ubuntu that is mode 640, owner root, group shadow. Check it:
ls -l /etc/shadow
If shadow is ever world-readable, every hash on the machine can be copied and attacked offline. Fix the permissions immediately. Ubuntu 22.04 and later hash passwords with yescrypt by default, a memory-hard algorithm that makes offline guessing expensive. That is a good default; do not weaken it.
1Review the account list
Read passwd with an auditor's eye. Each line has seven colon-separated fields; the ones that matter here are the username, the UID, and the shell:
awk -F: '$3 == 0 {print $1}' /etc/passwd
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $7}' /etc/passwd
getent group sudo
sudo awk -F: '$2 == "" {print $1}' /etc/shadow
The first command lists every account with UID 0. Only root should appear; a second UID 0 account is a backdoor. The second lists accounts with a real login shell; system accounts should end in /usr/sbin/nologin or /bin/false. The third shows who has full sudo through the sudo group. The fourth finds accounts with an empty password field, which on many configurations means login without any password. Investigate every hit.
2Lock, expire or remove unused accounts
For an account nobody should use, you have three strengths of removal. sudo passwd -l testuser locks the password by prefixing the hash with ! in shadow; sudo usermod -L testuser does the same. But locking blocks password logins only: SSH key authentication never consults the password hash, so a locked account with an authorized key still logs in. To stop all logins, expire the account:
sudo chage -E 1 testuser
sudo chage -E -1 testuser
chage -E 1 sets the expiry date to the epoch, so the account is expired; with PAM enabled, sshd refuses it even with a valid key. chage -E -1 clears the expiry. For a permanent removal, sudo userdel -r testuser deletes the account and its home directory; check for files owned by that UID elsewhere first, because they become orphaned.
Walk one account through those commands below and watch which state each one actually changes:
3Age passwords and audit logins
For human accounts that stay, set aging with chage. sudo chage -M 90 -W 14 learner forces a password change after 90 days and warns 14 days ahead. sudo chage -l learner shows the current policy; typical output (yours will differ):
Last password change : Nov 10, 2026
Password expires : Feb 08, 2027
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 90
Number of days of warning before password expires : 14
Apply aging to interactive human accounts, not to service accounts: an expired service password breaks automation at 3 a.m. For password quality, install libpam-pwquality and set minimum length and character classes in /etc/security/pwquality.conf; PAM then rejects weak choices at change time.
Finally, look at who actually logs in. who shows current sessions, last reads the login history from /var/log/wtmp, and sudo lastb reads failed attempts from /var/log/btmp. lastlog summarises the last login per account. An account that has never logged in, or one showing logins from strange addresses at strange hours, is a finding. On Raspberry Pi OS Bookworm the auth trail lives in the journal; you will read it in the logging section.
SSH and sudo: the two doors you must lock
5 min readS07B built your SSH foundation: host keys, ed25519 user keys, a first hardening policy. Here you consolidate it. The board policy is keys-only, no root login, and an explicit allowlist. Put it in a drop-in so package updates do not fight you: on Ubuntu, /etc/ssh/sshd_config starts with Include /etc/ssh/sshd_config.d/*.conf, and for most directives the first obtained value wins, so a drop-in there takes effect:
# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
MaxAuthTries 3
LoginGraceTime 30
AllowGroups sshusers
MaxAuthTries 3 cuts the guesses per connection (default 6). LoginGraceTime 30 closes connections that do not authenticate within 30 seconds (default 120), slowing connection-hogging attacks. AllowGroups sshusers admits only members of a dedicated group; AllowUsers learner is the per-user alternative and accepts patterns like learner@192.168.50.* to bind a user to a source network. Create the group and add yourself before you rely on it:
sudo groupadd sshusers
sudo usermod -aG sshusers learner
1Prove the key, then flip the policy
Order matters more than any single option. First install your public key with ssh-copy-id and prove a fresh key-only login from a second terminal, exactly as in S07B. Keep your current session open. Then validate and apply:
sudo sshd -t
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|allowgroups'
sudo systemctl restart ssh
sshd -t checks syntax and host keys, printing nothing on success. sshd -T dumps the effective configuration in lower case after Include and Match processing. Close the old session only after a new login succeeds. Flipping PasswordAuthentication no before the key works locks you out; on a headless board that means a serial console rescue.
A brute-force attack is automated guessing of credentials, usually thousands of password attempts against a login service. against SSH is constant background noise on any public address. Keys-only authentication makes password guessing useless, but the attempts still consume CPU and fill logs. fail2ban answers that. It watches the auth log and temporarily bans addresses that fail repeatedly. A In fail2ban, a jail is one watched service plus its filter, thresholds and ban action. is configured per service. Install it and configure the sshd jail in /etc/fail2ban/jail.local, never in jail.conf, which package upgrades overwrite:
# /etc/fail2ban/jail.local
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 5
findtime = 10m
bantime = 1h
ignoreip = 127.0.0.1/8 ::1 192.168.50.10
Five failures within ten minutes earn a one-hour ban. ignoreip must include the address you administer from, or you will ban yourself. With backend = systemd the jail reads the journal directly, which suits Ubuntu 24.04 and the Pi. Restart fail2ban, then check:
sudo systemctl restart fail2ban
sudo fail2ban-client status sshd
Watch: the jail tails the sshd log or journal for lines matching its filter, such as Failed password and Invalid user.
Count: each failure from one address increments a counter inside the findtime window.
Trigger: when the counter reaches maxretry, the jail fires its ban action.
Ban: the action inserts a firewall rule, iptables or nftables depending on setup, that drops packets from the address.
Expire: after bantime the rule is removed automatically; fail2ban-client set sshd unbanip removes it early.
To verify a ban, generate failures from a second machine and watch fail2ban-client status sshd list the banned address. With password authentication off, clients fail at method negotiation instead of sending a password; recent sshd filters match some of those preauth disconnects too, but the cleanest demo is a short controlled window with password auth still on.
2Least-privilege sudo
sudo is the second door. The sudo group grants ALL=(ALL:ALL) ALL, which is full root for every member. Audit who holds it with getent group sudo, and audit what any user may run with sudo -l (or sudo -ll for the long form). For an operator who only restarts one service and reads its logs, grant exactly that in a dedicated file:
sudo visudo -f /etc/sudoers.d/ops
ops ALL=(root) /usr/bin/systemctl restart sensor-collector
ops ALL=(root) /usr/bin/journalctl -u sensor-collector --no-pager
visudo syntax-checks on save, which is why you never edit sudoers with a plain editor: one typo can break sudo for everyone. Files in /etc/sudoers.d must be mode 0440, and sudo skips files whose names contain a dot or end in ~. Grant exact commands with fixed arguments. Avoid NOPASSWD except for tightly scoped automation, and remember it removes the one check that a stolen session still faces.
Going deepersudoers matching: last match wins, wildcards are dangerous
Unlike sshd, where the first value wins, sudoers applies rules in order and the last matching rule decides. A broad rule after a narrow one silently overrides it, so audit with sudo -l as the target user. Arguments match shell-style globs over the whole argument string, and * matches spaces too: allowing /usr/bin/find * lets the user append -exec /bin/sh \; and own root. Never grant editors, shells, pagers or anything with an exec escape; sudo vim is sudo sh in disguise. For an audit trail, sudo logs to syslog by default (auth.log on Ubuntu, the journal on the Pi); Defaults logfile="/var/log/sudo.log" adds a dedicated file, and Defaults log_input,log_output records session I/O for replay with sudoreplay.
Filesystem security: permissions, flags and audits
4 min readS03A taught you rwx, the special bits and umask. Hardening turns that knowledge into audits. Start with defaults: a umask of 027 means new files are 640 and new directories 750, so other users get nothing. Set it system-wide through UMASK 027 in /etc/login.defs, which pam_umask applies at login, or in a script under /etc/profile.d/. Check the live value with umask.
1Audit world-writable files
A world-writable file can be changed by any local account, which turns a small compromise into a big one. Find them, staying on the root filesystem with -xdev so you do not wander into proc, sys or network mounts:
sudo find / -xdev -type f -perm -002 -ls
sudo find / -xdev -type d -perm -002 ! -perm -1000 -ls
The first lists world-writable files. The second lists world-writable directories that lack the sticky bit; without sticky, any user can delete or rename anyone else's files there. /tmp itself is world-writable by design and carries sticky (mode 1777), so it will not appear in the second list. For every hit, decide: tighten the mode, add sticky, or move the file to a proper location.
2Audit SUID and SGID programs
A set-user-ID root program runs with root's effective UID no matter who starts it, so every one is a potential privilege-escalation target. List the whole set and save it as a baseline:
sudo find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -ls | tee /root/suid-baseline.txt
-perm -4000 matches setuid, -perm -2000 setgid. Stock Ubuntu shows a small, well-known set: passwd, su, sudo, mount, umount, newgrp, chfn, chsh, gpasswd, ssh-keysign. Anything else deserves a question: who installed it, and does it need the bit? Reduce the set by removing unused packages, or with sudo chmod u-s /path/to/binary for a program you keep but never need privileged. Do not strip passwd or sudo; you would break password changes and administration. Re-run after every upgrade and diff against the baseline: upgrades restore bits and new packages add programs.
3Protect critical files with chattr
Permissions can be changed by root, and a compromised root-level process changes them freely. The immutable flag is an inode flag that forbids all modification of a file, including writes, deletion, renaming and permission changes, until the flag is cleared. goes further: with it set, even root cannot modify the file. Try it on a scratch file:
echo "critical setting" > /home/learner/vault.conf
sudo chattr +i /home/learner/vault.conf
lsattr /home/learner/vault.conf
rm /home/learner/vault.conf
sudo rm /home/learner/vault.conf
sudo chattr -i /home/learner/vault.conf
lsattr prints the inode flags as a row of letters; an i in that row means immutable. Both removes fail with "Operation not permitted", even under sudo, until chattr -i clears the flag. The related +a flag makes a file append-only, which suits log files that must never be rewritten. These flags live in the inode, not in the permission bits, and they work on ext2, ext3, ext4, XFS, Btrfs and f2fs. They do not work on tmpfs, proc, sysfs, or the Windows drives mounted in WSL, so test on the board's ext4 root or a VM.
Going deeperWhat immutable really blocks, and who can clear it
The kernel enforces FS_IMMUTABLE_FL on write, truncate, unlink, link, rename, chmod, chown, utime and extended-attribute changes, checked in the VFS and filesystem paths before the operation proceeds. Setting or clearing the flag requires CAP_LINUX_IMMUTABLE, so root can lift it but a compromised daemon without that capability cannot. From C, use ioctl(fd, FS_IOC_GETFLAGS) and FS_IOC_SETFLAGS. Two operational traps: an immutable file under /etc or /usr makes apt or dpkg fail with "Operation not permitted", so never blanket-apply +i across system directories; and an immutable /etc/shadow silently breaks passwd and chage. Protect a short, documented list, and clear it before maintenance.
Before running apt upgrade on a hardened board, check for immutable files in the paths the upgrade touches: sudo lsattr -R /etc 2>/dev/null | grep -- '-i-'. Clear the flag, upgrade, then re-apply and record it. An upgrade that fails halfway because of one protected file is a self-inflicted outage.
Updates: CVEs and automatic patching
3 min readA A CVE identifier, such as CVE-2021-44228, is a public catalogue number for one disclosed security vulnerability. names a specific flaw so everyone talks about the same bug. MITRE has run the CVE programme since 1999; the National Vulnerability Database adds severity scores. The lifecycle drives your planning: a researcher reports a bug, the vendor fixes it, the fix and CVE are published, maintainers backport it into stable packages, and your machine is safe only once it installs that package. Attackers read the same publications and scan within hours. Your exposure window is the time you run the unpatched version.
On Ubuntu, security fixes arrive through the -security pocket, for example noble-security on 24.04. Check what is pending and apply it:
sudo apt update
apt list --upgradable
sudo apt upgrade
apt list --upgradable shows candidate versions; security updates are the ones you must not defer. sudo apt autoremove clears orphaned dependencies, including old kernels. Read its list before confirming: on a board, keep one known-good kernel as a fallback, because a bad update plus no fallback means a physical rescue.
1Enable unattended-upgrades
Doing this by hand every week fails exactly when you are busy. The unattended-upgrades package installs security updates automatically. Ubuntu ships it; verify it is enabled:
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
The reconfigure step writes /etc/apt/apt.conf.d/20auto-upgrades, which turns on the periodic apt behaviour:
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
The policy lives in /etc/apt/apt.conf.d/50unattended-upgrades. Its Unattended-Upgrade::Allowed-Origins list decides what may be installed automatically; the default admits only the security pocket:
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}-security";
};
That default is the right one for a board: security fixes only, no feature upgrades that could change behaviour mid-deployment. Two systemd timers drive the work: apt-daily.timer refreshes lists and downloads, apt-daily-upgrade.timer installs. Inspect them with systemctl list-timers apt-daily*. Before trusting the setup, dry-run it:
sudo unattended-upgrade --dry-run --debug
The dry run prints which packages would be upgraded and why, without changing anything. Results land in /var/log/unattended-upgrades/.
2Plan the reboot, not just the patch
Some fixes, especially kernel and libc updates, take effect only after a restart. The config offers Unattended-Upgrade::Automatic-Reboot "true" with Automatic-Reboot-Time "02:00". On a server that is reasonable; on a sensor node it is a decision: a reboot interrupts capture and adds flash write cycles. Prefer a planned maintenance window, or the A/B image updates with rollback that later embedded modules build. And after apt autoremove, keep one known-good kernel as a recovery path.
A Buildroot or BusyBox image has no apt at all. There, patching means rebuilding the image with updated package versions and flashing it, which is why such products pair a read-only root filesystem with a signed update mechanism. The principle is identical: shrink the window between a published fix and a running fix. The mechanics move from the package manager to the build system.
Kernel hardening knobs with sysctl
5 min readThe kernel exposes hundreds of tunables under /proc/sys. The sysctl command reads and writes them by dotted name: kernel.randomize_va_space is the file /proc/sys/kernel/randomize_va_space. Read the security-relevant ones first:
sysctl kernel.randomize_va_space kernel.kptr_restrict kernel.dmesg_restrict
sysctl net.ipv4.ip_forward kernel.sysrq
ASLR, address-space layout randomisation, places a process's stack, libraries and other regions at unpredictable addresses each run. is controlled by kernel.randomize_va_space: 0 disables it, 1 randomises the stack, shared libraries and the vdso, and 2 also randomises the brk heap. The value should be 2, and on Ubuntu it is. ASLR forces an exploit to guess addresses instead of hard-coding them, which turns many memory-corruption attacks from reliable into probabilistic. kernel.kptr_restrict hides kernel pointers from unprivileged readers of proc files when set to 1 or higher; kernel.dmesg_restrict restricts the kernel log buffer to privileged readers when set to 1, which is why Ubuntu asks for sudo dmesg. Both remove information that helps an attacker build a kernel exploit.
net.ipv4.ip_forward decides whether the machine routes packets between interfaces. A sensor node is not a router; the value should be 0 unless you deliberately built a gateway in S07B. An unintended forwarder can bridge networks you meant to keep apart.
kernel.sysrq controls the magic System Request keys, which can force a sync, remount filesystems read-only, or reboot even when userspace is hung. The value is a bitmask: 0 disables all, 1 enables all, and subsets use 16 for sync, 32 for read-only remount and 128 for reboot and power-off. Many installs leave a safe subset enabled. For an unattended board, decide explicitly: 0 removes a local attack avenue, while sync and read-only remount can still save a filesystem during a hang. Read yours, choose, document.
Toggle the bits below and watch the decimal total that you would write into the sysctl file:
1Make the settings persistent
A runtime write is lost at reboot. This is the single most common sysctl mistake:
sudo sysctl -w kernel.sysrq=0
That changes the running kernel only. Persistence lives in configuration files. Create a drop-in:
# /etc/sysctl.d/99-hardening.conf
kernel.randomize_va_space = 2
kernel.kptr_restrict = 1
kernel.dmesg_restrict = 1
net.ipv4.ip_forward = 0
kernel.sysrq = 0
Then load every configured file and verify:
sudo sysctl --system
sysctl kernel.sysrq
sysctl --system reads the standard directories, including /etc/sysctl.d, in order, so your 99-prefixed file applies after the distribution defaults. At boot, systemd-sysctl applies the same files automatically. The next lesson goes deeper into sysctl, sysfs and the rest of the kernel's admin interface.
Going deeperWhat ASLR actually randomises, and its limits
With randomize_va_space=2 the kernel randomises the stack top, the mmap base (which places shared libraries), the vdso, the brk start, and the load address of PIE executables. A non-PIE executable loads at a fixed text address, which is why Ubuntu's gcc builds PIE by default: without PIE, ASLR protects the stack but leaves the code where the attacker expects it. Entropy is smaller on 32-bit architectures, so brute-forcing a 32-bit ARM layout is far more practical than a 64-bit one; prefer 64-bit userspace on capable boards. For debugging, disable ASLR per process with setarch $(uname -m) -R ./program and inspect /proc/PID/maps. ASLR is a mitigation, not a boundary: it raises exploitation cost but does not fix the bug, so patching and confinement stay in the stack.
WSL2 shares a Microsoft-built kernel across distributions. Some knobs are read-only, absent, or behave differently, and writes may fail. Treat WSL as a place to read and discuss these settings, and do the write-and-reboot verification in a VM or on the Pi.
Logging, auditing and detection
4 min readOn Ubuntu 24.04 with systemd, the journal is the centre of gravity. sshd, sudo and cron all write to it. Read it by unit, by program, or by priority:
journalctl -u ssh --since today --no-pager
journalctl _COMM=sshd -p warning -b --no-pager
-u ssh follows the SSH unit, _COMM=sshd matches the process name, -p warning filters severity, and -b limits to the current boot. Ubuntu also runs rsyslog, which writes traditional files. Its auth log uses one ISO 8601 timestamp field, so a line looks like 2026-11-16T04:45:42.918293+00:00 labpc sshd[1234]: Failed password for learner from 192.0.2.10 port 51234 ssh2. Older releases used a three-field "Nov 16 04:45:42" prefix; do not write parsers against that old shape on 24.04.
1Hunt brute force in the auth log
You already own the log-analysis tools from S04B. Point them at authentication failures:
sudo grep -E 'Failed password|Invalid user' /var/log/auth.log \
| grep -oE 'from ([0-9]{1,3}\.){3}[0-9]{1,3}' \
| sort | uniq -c | sort -rn | head
This extracts the source address from each failure line, counts occurrences, and ranks them. A long tail of addresses with a handful of attempts each is internet background noise. One address with hundreds of attempts is a targeted guesser, and fail2ban should already have banned it. Repeated "Invalid user" lines for names like admin, test or oracle show dictionary scanning. On Raspberry Pi OS Bookworm there is no /var/log/auth.log; the same trail lives in the journal, so run journalctl -u ssh and grep that instead. In WSL without rsyslog, the journal is also your only source.
Check what that pattern really captures before trusting the counts:
2Add auditd for file and syscall evidence
The journal records what services say about themselves. The kernel audit subsystem records what actually happened, per rules you write, independent of the service's own logging. Install it and add a watch:
sudo apt install auditd
sudo auditctl -w /etc/passwd -p wa -k identity
sudo auditctl -w /etc/shadow -p wa -k identity
sudo auditctl -w /etc/sudoers -p wa -k privilege
-w watches a path, -p wa triggers on writes and attribute changes, and -k tags records with a key you can search by. Runtime rules vanish at reboot, so persist them in /etc/audit/rules.d/identity.rules with the same lines, then load them with sudo augenrules --load. Search and report:
sudo ausearch -k identity -i
sudo aureport --auth
ausearch -i interprets numeric UIDs and syscalls into names; records land in /var/log/audit/audit.log. A watch on passwd, shadow and sudoers answers the question "did anyone change the accounts or the privilege rules?" with kernel-level evidence. Keep rules few and targeted on a board: every rule costs CPU and log writes, and flash wears out. Note that containers and WSL may lack the kernel audit interface; use a VM or the Pi for this part.
3Summarise and watch file integrity
For a daily summary, logwatch parses the logs and mails a report; sudo logwatch --range today --detail high prints one on demand. logcheck mails only security-relevant lines. For tamper detection, AIDE hashes a baseline of critical files and reports changes: initialise the database (on Debian and Ubuntu the aideinit wrapper does this), then run sudo aide --check periodically. After a legitimate change, rebuild with sudo aide --update and promote the new database, or later checks drown in known-good differences. Store the baseline where an attacker cannot rewrite it, ideally off the board. AIDE detects change; it does not prevent it, so it sits behind chattr and permissions, not instead of them.
Capabilities, confinement and mandatory access control
5 min readTraditional UNIX privilege is all-or-nothing: effective UID 0 bypasses every check. That is why a bug in any set-user-ID root program is a full compromise. Linux A capability is one unit of root privilege, such as binding a low port or changing the system clock, that the kernel checks independently of the user ID. split that power into around forty units. A process can hold just the ones it needs. The classic example is binding a port below 1024, which normally requires root: a custom web server needs CAP_NET_BIND_SERVICE for port 443, not the power to read shadow or load modules.
File capabilities attach units to an executable, stored in the security.capability extended attribute. The tools are setcap and getcap, provided by libcap (the libcap2-bin package on Ubuntu). A real example already on your system is ping, which needs raw sockets:
getcap /usr/bin/ping
cp /usr/bin/ping /home/learner/ping-test
sudo setcap cap_net_raw=+ep /home/learner/ping-test
getcap /home/learner/ping-test
/home/learner/ping-test -c1 127.0.0.1
Typical output of the first command (yours will differ): /usr/bin/ping cap_net_raw=ep. The =ep sets the file's permitted and effective sets: permitted means the process may use the capability, effective means it is active at once. A capability-aware program can take =p instead and raise the capability only when needed, which is tighter. Two traps: a package upgrade replaces the binary and drops its capabilities, and a nosuid mount ignores file capabilities entirely. For services, systemd is the cleaner route: AmbientCapabilities=CAP_NET_BIND_SERVICE grants the unit at start, and CapabilityBoundingSet= caps what the service can ever hold. Inspect a process with grep Cap /proc/PID/status, or getpcap $$ for your shell.
Read: the kernel reads the file's security.capability attribute during exec.
Permitted: the file's permitted set is added to the process's new permitted set, masked by the bounding set.
Effective: if the file's effective bit is set, the new permitted capabilities are also made effective at once.
Inheritable: the file's inheritable set is ANDed with the process's inheritable set, adding only what both sides allow.
Run: the program starts with exactly those units, not full root, and a capability-aware program can drop more.
Capabilities shrink what a process may do; confinement shrinks what it may see and reach. A A namespace isolates one kind of system view, such as the process list, mount tree or network stack, so a process sees only its own slice. is the kernel mechanism behind containers: the next lesson builds containers from namespaces plus cgroups. seccomp is a kernel filter that allows a process to call only a whitelisted set of system calls, killing or erroring on anything else. goes further and restricts the syscall surface itself; systemd exposes it as SystemCallFilter=, and container runtimes ship default profiles. Together with NoNewPrivileges=true, which blocks gaining privilege through exec, these give a service a small, fixed box to live in.
1Mandatory access control with AppArmor
Permissions and capabilities are still decided by owners and by root; mandatory access control is a system-wide policy, enforced by the kernel, that restricts programs even when they run as root (MAC) is the layer above. SELinux, released by the NSA in 2000 and merged into Linux 2.6.0, labels every file and process and enforces rules over those labels; it is the RHEL and Fedora default, and getenforce reports enforcing, permissive or disabled. AppArmor, mainlined in 2.6.36, confines programs with path-based profiles and is the Ubuntu default. Check what is loaded:
sudo aa-status
You will see profiles in enforce mode and some in complain mode. Enforce blocks violations; complain logs them without blocking, which is how you tune a new profile. To confine your own service, write a profile named after its binary path, start in complain mode, and watch the journal for apparmor="DENIED" lines:
# /etc/apparmor.d/usr.local.bin.sensor-collector
#include <tunables/global>
/usr/local/bin/sensor-collector {
#include <abstractions/base>
/usr/local/bin/sensor-collector ix,
/etc/sensor-collector.conf r,
/var/lib/sensor-collector/** rw,
/dev/ttyUSB0 rw,
network inet stream,
}
Load it with sudo apparmor_parser -r /etc/apparmor.d/usr.local.bin.sensor-collector, switch modes with aa-complain and aa-enforce, and re-check aa-status. Each rule grants one access: read a config, read-write a data directory, use the serial device, open TCP streams. Everything else is denied, even if the process runs as root. The later security module builds this out with SELinux, seccomp profiles and secure provisioning.
The WSL2 kernel is built without AppArmor enabled, so aa-status reports it as disabled even though the userspace tools exist. That is a kernel build choice, not a broken installation. Run the profile exercises in a VM or on the Pi, where Ubuntu enables AppArmor by default.
Common mistakes and the hardening checklist
4 min readThese are the mistakes that appear in real hardening work, with symptom, cause, diagnosis and fix:
- Locked out of SSH. Symptom: "Permission denied (publickey)" on every login. Cause: keys-only applied before a working key existed, or the user missing from AllowUsers/AllowGroups. Diagnose from the console or an open session:
sudo sshd -T, check authorized_keys content and modes (700 directory, 600 file). Fix from the console, then re-apply key first, policy second. - sudo broken for everyone. Symptom: "sudo: parse error in /etc/sudoers" or "no valid sudoers sources found". Cause: a plain-editor typo, or a sudoers.d file with wrong mode or a dot in its name. Recover with
pkexec visudoor a root shell; always edit through visudo, which validates on save. - apt upgrade fails with "Operation not permitted". Symptom: dpkg cannot overwrite one config file. Cause: a leftover
chattr +i. Diagnose withsudo lsattron the named path; clear the flag, upgrade, re-apply and document. - fail2ban banned your own address. Symptom: your management station suddenly cannot connect. Cause: your IP missing from ignoreip. Diagnose from the console with
sudo fail2ban-client status sshd; unban withsudo fail2ban-client set sshd unbanip YOUR_IP, then add the address to ignoreip. - Port still open after disabling a service. Symptom:
ss -tulpnstill shows the listener. Cause: socket activation or a second instance. Checksystemctl list-units --type=socketand the ss process column; disable the socket too, then re-check. - A setting vanished after reboot. Symptom: sysctl values, audit rules or fail2ban tweaks reset. Cause: runtime-only changes. Persist in
/etc/sysctl.d,/etc/audit/rules.dandjail.local, then reload and reboot-test.
1Run a baseline audit with Lynis
Checklists keep humans honest; tools keep checklists complete. The CIS Benchmarks are free, consensus-built hardening guides per distribution, and they define what "hardened" means in measurable terms. Lynis is the open-source auditor from CISOfy that checks hundreds of such items:
sudo apt install lynis
sudo lynis audit system
Lynis prints findings with test identifiers and writes /var/log/lynis.log plus /var/log/lynis-report.dat, ending with a hardening index. Treat the score as a thermometer, not a target: a high score bought by disabling logging is a worse system. Read each suggestion, judge it against your threat model, apply it, re-run. On the Pi expect SSH, service and password findings; on WSL expect kernel tests to be skipped, because the kernel is not yours to configure.
2Keep the loop, and add the board items
Hardening is a loop, not a project: baseline audit, fix, re-audit, document, monitor. Re-audit after every image change and package upgrade, and diff your SUID and world-writable baselines at the same time. For embedded products add what server checklists miss: no factory default credentials (Mirai's whole business model), no password SSH, the serial console treated as a trust boundary, a read-only root filesystem where possible, signed updates with rollback, and persistent logging so an incident survives reboot. Each maps to a mechanism you now know.
Try it
1 min readThe lab pack turns this lesson into three exercises on your Pi: run Lynis, apply a chosen set of fixes, and re-audit to prove the change; audit SUID and world-writable files with find, reduce the set, and demonstrate chattr +i blocking even root; then configure the fail2ban sshd jail and verify a real ban from a second machine. Keep your console access open throughout, and write down every change so you can undo it. When you finish, you should be able to hand someone your hardening notes and have them reproduce the exact state of the board.
Key takeaways
- Hardening is layers: every listening port, account and writable file is surface you can shrink.
- Install and prove an SSH key before you disable passwords, or you lock yourself out.
- passwd -l locks a password, not the account; SSH keys still work until you expire the account.
- sudoers rules use last-match-wins and shell-style wildcards; grant exact commands, not editors or shells.
- chattr +i blocks writes, deletes and chmod even for root, until the flag is cleared.
- unattended-upgrades installs security patches on a timer; a board still needs a planned reboot window.
- sysctl -w is runtime only; persistence lives in /etc/sysctl.d and is loaded at boot.
- Capabilities split root into small units; AppArmor and SELinux confine programs even when root runs them.
Reflect
- Explain to a colleague, in two sentences: explain attack surface, least privilege and defence in depth, and map a host's surface with ss and systemctl.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Man pagesshd_config(5): OpenSSH daemon configuration file
- Man pagesudoers(5): default sudo security policy plugin
- Man pagecapabilities(7): overview of Linux capabilities
- Official docsLinux kernel documentation: /proc/sys/kernel
- Official docsLinux kernel documentation: Linux Magic System Request Key Hacks
- Official docsDebian Wiki: UnattendedUpgrades
- Official docsAppArmor project wiki
- Official docsLynis documentation, CISOfy
Official documentation first. Links checked when the lesson was published.