Module 1 · Lesson 21 of 28

Security hardening

Harden a Linux system: shrink the attack surface, lock down SSH and sudo, audit files, patch automatically, and verify with Lynis.

  • Beginner to advanced
  • 30 min read
  • 55 min lab
What you will need

Ubuntu 24.04 (WSL2 with systemd enabled, VM or container) with sudo rights; a Raspberry Pi reachable over SSH for the board labs

By the end you can 7 objectives
  • Explain attack surface, least privilege and defence in depth, and map a host's surface with ss and systemctl.
  • Audit user accounts with passwd, shadow, chage, last and who, and disable or expire accounts safely.
  • Apply an SSH keys-only policy with AllowUsers and fail2ban, and write least-privilege sudoers rules audited with sudo -l.
  • Audit SUID, SGID and world-writable files with find, and protect critical files with chattr +i.
  • Configure unattended-upgrades and persistent sysctl hardening, and explain what each knob changes.
  • Evaluate capabilities, namespaces, seccomp, SELinux and AppArmor as confinement layers.
  • Run a Lynis audit, interpret its findings, and judge a hardening change by re-auditing rather than by score alone.
Read the lesson below; the video will appear here when published.
01

Why harden: the attacker's arithmetic

3 min read

Every Linux system ships with choices made for convenience, not for your threat model. Services listen on the network. Accounts exist that nobody uses. Programs run with more power than they need. Hardening is the deliberate work of removing unnecessary access and limiting the damage when something is compromised. It is not one tool or one switch. It is a habit you apply to every layer of the system.

The stakes are concrete. In November 1988 the Morris worm spread across the early internet by exploiting a debug option in sendmail and weak passwords. It infected thousands of machines and showed that a network-facing service is a target the moment it exists. In 2016 the Mirai botnet scanned the internet for embedded devices, mostly cameras and routers, and logged in with factory default usernames and passwords over Telnet and SSH. Hundreds of thousands of boards were enslaved and used to knock large websites offline. The lesson for you is direct: a Raspberry Pi with SSH open and a weak password is found by automated scanners within hours, not weeks.

Three ideas organise everything in this lesson. The is what you shrink first. is what you apply to everything that remains. is why you keep going after the obvious fixes: a firewall, then keys-only SSH, then tight sudo rules, then file permissions, then confinement, then logging. An attacker must beat every layer. You only need one layer to hold.

This lesson climbs from a read-only audit to expert tools: capabilities that replace set-user-ID root, and confinement that restricts even root. You own the building blocks already: users and permissions from S03A, SSH and firewalls from S07B, troubleshooting from S10B. Here you point them at one question: what can an attacker reach, and how do you make that smaller?

Embedded boards sharpen every point. A sensor gateway sits unattended in a field cabinet, often on a public address, with little RAM, flash that wears out, and a serial console anyone can plug into. Each section carries the board angle, and the labs run against your Pi.

Defence in depthNetwork firewalldefault deny, allow only needed portsService policykeys-only SSH, scoped sudoAccounts and filesleast privilege, tight permissionsConfinementcapabilities, AppArmor, seccompKernel knobsASLR, pointer hiding, sysrqLogs and auditdetect what prevention missedan attacker must beat every layerNetwork firewalldefault deny, allow onlyneeded portsService policykeys-only SSH, scoped sudoAccounts and filesleast privilege, tightpermissionsConfinementcapabilities, AppArmor,seccompKernel knobsASLR, pointer hiding, sysrqLogs and auditdetect what prevention missedan attacker must beat everylayer
Defence in depth: independent layers from the network edge down to kernel knobs, each assuming the layer above may fail.
02

Map and shrink the attack surface

3 min read

Start with the network. A listening socket, a network endpoint where a service accepts incoming connections on a port, is a door. The tool ss lists them. Run it with sudo so it can name the owning process of every socket, not just yours:

bash
sudo ss -tulpn

The flags mean: -t TCP, -u UDP, -l listening only, -p show the process, -n show numbers instead of resolving names. Typical output (yours will differ):

text
Netid State  Local Address:Port  Process
tcp   LISTEN 0.0.0.0:22          users:(("sshd",pid=812,fd=3))
tcp   LISTEN [::]:22            users:(("sshd",pid=812,fd=4))
udp   UNCONN 127.0.0.1:323      users:(("chronyd",pid=640,fd=5))
tcp   LISTEN *:8080             users:(("python3",pid=2310,fd=3))

The local address column decides who can knock. 0.0.0.0:22 and [::]:22 mean SSH accepts connections on every interface. 127.0.0.1:323 is chrony's command port bound to loopback: only local processes can reach it. *:8080 is a development server exposed to the world. The rule: a loopback bind is local-only; a wildcard bind is exposed. For each exposed listener ask: does this need to be reachable from outside? If not, remove it or bind it to loopback.

1Inventory enabled services

Next list what starts at boot. A service that is enabled but not currently running is still surface, because it will listen after the next reboot:

bash
systemctl list-unit-files --state=enabled
systemctl list-units --type=service --state=running

A full image enables printing (cups), Bluetooth, avahi mDNS discovery and more; on a headless board most are pointless. Disable and stop what you do not need:

bash
sudo systemctl disable --now cups
sudo systemctl disable --now avahi-daemon

disable removes the start-at-boot links; --now also stops the running instance. Verify with ss -tulpn that the port is gone. If it remains, the service may be socket-activated: systemd holds the listener itself, as with ssh.socket in S07B. Check systemctl list-units --type=socket and disable the socket too. systemctl mask is stronger than disable: it links the unit to /dev/null so nothing can start it, even as a dependency.

2Count accounts and close the network edge

Accounts are surface too. Every account with a password, a shell and SSH eligibility is a login an attacker can guess at. You will audit them in the next section. At the network edge, keep the default-deny firewall you built in S07B: allow only the ports you deliberately expose, drop the rest. With ufw the shape is sudo ufw default deny incoming, then explicit allows such as sudo ufw allow 22/tcp. A firewall does not make a weak service safe; it hides the services you forgot about. Both matter.

On the Pi

Raspberry Pi OS enables a few services you may not want on a headless sensor node. Run the same inventory over SSH. Keep ssh, and consider whether you need anything else at all. A board that only reads sensors and uploads data often needs exactly one listening port.

What to inventoryAttack surfaceListening portsss -tulpnEnabled servicessystemctl list-unit-filesUser accountspasswd, sudo group, keysRisky filesSUID, world-writableAttack surfaceListening portsss -tulpnEnabled servicessystemctl list-unit-filesUser accountspasswd, sudo group, keysRisky filesSUID, world-writable
The four kinds of attack surface to inventory: listening ports, enabled services, user accounts and writable or privileged files.
03

Account security: passwords, aging and audits

5 min read

You met /etc/passwd and /etc/shadow in S03A. Recall the split: passwd holds account metadata and is world-readable; shadow holds password hashes and is readable only by root and the shadow group. On Ubuntu that is mode 640, owner root, group shadow. Check it:

bash
ls -l /etc/shadow

If shadow is ever world-readable, every hash on the machine can be copied and attacked offline. Fix the permissions immediately. Ubuntu 22.04 and later hash passwords with yescrypt by default, a memory-hard algorithm that makes offline guessing expensive. That is a good default; do not weaken it.

1Review the account list

Read passwd with an auditor's eye. Each line has seven colon-separated fields; the ones that matter here are the username, the UID, and the shell:

bash
awk -F: '$3 == 0 {print $1}' /etc/passwd
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $7}' /etc/passwd
getent group sudo
sudo awk -F: '$2 == "" {print $1}' /etc/shadow

The first command lists every account with UID 0. Only root should appear; a second UID 0 account is a backdoor. The second lists accounts with a real login shell; system accounts should end in /usr/sbin/nologin or /bin/false. The third shows who has full sudo through the sudo group. The fourth finds accounts with an empty password field, which on many configurations means login without any password. Investigate every hit.

2Lock, expire or remove unused accounts

For an account nobody should use, you have three strengths of removal. sudo passwd -l testuser locks the password by prefixing the hash with ! in shadow; sudo usermod -L testuser does the same. But locking blocks password logins only: SSH key authentication never consults the password hash, so a locked account with an authorized key still logs in. To stop all logins, expire the account:

bash
sudo chage -E 1 testuser
sudo chage -E -1 testuser

chage -E 1 sets the expiry date to the epoch, so the account is expired; with PAM enabled, sshd refuses it even with a valid key. chage -E -1 clears the expiry. For a permanent removal, sudo userdel -r testuser deletes the account and its home directory; check for files owned by that UID elsewhere first, because they become orphaned.

Walk one account through those commands below and watch which state each one actually changes:

passwd -l or usermod -L only reaches password-locked, where an SSH key still logs in; chage -E 1 expires the account, which PAM refuses even with a valid key, and chage -E -1 returns it to active; userdel -r removes it, and a later useradd -m creates a different account with a new UID rather than restoring the old one.

  • states: [{"id": "active", "label": "Active"}, {"id": "locked", "label": "Password-locked"}, {"id": "expired", "label": "Expired"}, {"id": "removed", "label": "Removed"}]
  • transitions: [{"from": "active", "to": "locked", "event": "passwd -l"}, {"from": "locked", "to": "active", "event": "passwd -u"}, {"from": "active", "to": "expired", "event": "chage -E 1"}, {"from": "locked", "to": "expired", "event": "chage -E 1"}, {"from": "expired", "to": "active", "event": "chage -E -1"}, {"from": "active", "to": "removed", "event": "userdel -r"}, {"from": "locked", "to": "removed", "event": "userdel -r"}, {"from": "expired", "to": "removed", "event": "userdel -r"}, {"from": "removed", "to": "active", "event": "useradd -m"}]

3Age passwords and audit logins

For human accounts that stay, set aging with chage. sudo chage -M 90 -W 14 learner forces a password change after 90 days and warns 14 days ahead. sudo chage -l learner shows the current policy; typical output (yours will differ):

text
Last password change                                    : Nov 10, 2026
Password expires                                        : Feb 08, 2027
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 90
Number of days of warning before password expires       : 14

Apply aging to interactive human accounts, not to service accounts: an expired service password breaks automation at 3 a.m. For password quality, install libpam-pwquality and set minimum length and character classes in /etc/security/pwquality.conf; PAM then rejects weak choices at change time.

Finally, look at who actually logs in. who shows current sessions, last reads the login history from /var/log/wtmp, and sudo lastb reads failed attempts from /var/log/btmp. lastlog summarises the last login per account. An account that has never logged in, or one showing logins from strange addresses at strange hours, is a finding. On Raspberry Pi OS Bookworm the auth trail lives in the journal; you will read it in the logging section.

Account lifecycleActivePassword lockedkeys still workExpiredall logins refusedRemovedTransitionspasswd -lActivePassword lockedkeys still workpasswd -uPassword lockedkeys still workActivechage -E 1ActiveExpiredall logins refusedchage -E -1Expiredall logins refusedActiveuserdel -rActiveRemovedActivePassword lockedkeys still workExpiredall logins refusedRemovedTransitionspasswd -lActivePasswordlockedkeys stillworkpasswd -uPasswordlockedkeys stillworkActivechage -E 1ActiveExpiredall loginsrefusedchage -E -1Expiredall loginsrefusedActiveuserdel -rActiveRemoved
Account lifecycle states: active, password-locked, expired and removed, with the commands that move between them.
04

SSH and sudo: the two doors you must lock

5 min read

S07B built your SSH foundation: host keys, ed25519 user keys, a first hardening policy. Here you consolidate it. The board policy is keys-only, no root login, and an explicit allowlist. Put it in a drop-in so package updates do not fight you: on Ubuntu, /etc/ssh/sshd_config starts with Include /etc/ssh/sshd_config.d/*.conf, and for most directives the first obtained value wins, so a drop-in there takes effect:

text
# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
MaxAuthTries 3
LoginGraceTime 30
AllowGroups sshusers

MaxAuthTries 3 cuts the guesses per connection (default 6). LoginGraceTime 30 closes connections that do not authenticate within 30 seconds (default 120), slowing connection-hogging attacks. AllowGroups sshusers admits only members of a dedicated group; AllowUsers learner is the per-user alternative and accepts patterns like learner@192.168.50.* to bind a user to a source network. Create the group and add yourself before you rely on it:

bash
sudo groupadd sshusers
sudo usermod -aG sshusers learner

1Prove the key, then flip the policy

Order matters more than any single option. First install your public key with ssh-copy-id and prove a fresh key-only login from a second terminal, exactly as in S07B. Keep your current session open. Then validate and apply:

bash
sudo sshd -t
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|allowgroups'
sudo systemctl restart ssh

sshd -t checks syntax and host keys, printing nothing on success. sshd -T dumps the effective configuration in lower case after Include and Match processing. Close the old session only after a new login succeeds. Flipping PasswordAuthentication no before the key works locks you out; on a headless board that means a serial console rescue.

against SSH is constant background noise on any public address. Keys-only authentication makes password guessing useless, but the attempts still consume CPU and fill logs. fail2ban answers that. It watches the auth log and temporarily bans addresses that fail repeatedly. A is configured per service. Install it and configure the sshd jail in /etc/fail2ban/jail.local, never in jail.conf, which package upgrades overwrite:

text
# /etc/fail2ban/jail.local
[sshd]
enabled  = true
port     = ssh
backend  = systemd
maxretry = 5
findtime = 10m
bantime  = 1h
ignoreip = 127.0.0.1/8 ::1 192.168.50.10

Five failures within ten minutes earn a one-hour ban. ignoreip must include the address you administer from, or you will ban yourself. With backend = systemd the jail reads the journal directly, which suits Ubuntu 24.04 and the Pi. Restart fail2ban, then check:

bash
sudo systemctl restart fail2ban
sudo fail2ban-client status sshd
Step by stepHow fail2ban bans an address
  1. Watch: the jail tails the sshd log or journal for lines matching its filter, such as Failed password and Invalid user.

  2. Count: each failure from one address increments a counter inside the findtime window.

  3. Trigger: when the counter reaches maxretry, the jail fires its ban action.

  4. Ban: the action inserts a firewall rule, iptables or nftables depending on setup, that drops packets from the address.

  5. Expire: after bantime the rule is removed automatically; fail2ban-client set sshd unbanip removes it early.

To verify a ban, generate failures from a second machine and watch fail2ban-client status sshd list the banned address. With password authentication off, clients fail at method negotiation instead of sending a password; recent sshd filters match some of those preauth disconnects too, but the cleanest demo is a short controlled window with password auth still on.

2Least-privilege sudo

sudo is the second door. The sudo group grants ALL=(ALL:ALL) ALL, which is full root for every member. Audit who holds it with getent group sudo, and audit what any user may run with sudo -l (or sudo -ll for the long form). For an operator who only restarts one service and reads its logs, grant exactly that in a dedicated file:

bash
sudo visudo -f /etc/sudoers.d/ops
text
ops ALL=(root) /usr/bin/systemctl restart sensor-collector
ops ALL=(root) /usr/bin/journalctl -u sensor-collector --no-pager

visudo syntax-checks on save, which is why you never edit sudoers with a plain editor: one typo can break sudo for everyone. Files in /etc/sudoers.d must be mode 0440, and sudo skips files whose names contain a dot or end in ~. Grant exact commands with fixed arguments. Avoid NOPASSWD except for tightly scoped automation, and remember it removes the one check that a stolen session still faces.

Going deepersudoers matching: last match wins, wildcards are dangerous

Unlike sshd, where the first value wins, sudoers applies rules in order and the last matching rule decides. A broad rule after a narrow one silently overrides it, so audit with sudo -l as the target user. Arguments match shell-style globs over the whole argument string, and * matches spaces too: allowing /usr/bin/find * lets the user append -exec /bin/sh \; and own root. Never grant editors, shells, pagers or anything with an exec escape; sudo vim is sudo sh in disguise. For an audit trail, sudo logs to syslog by default (auth.log on Ubuntu, the journal on the Pi); Defaults logfile="/var/log/sudo.log" adds a dedicated file, and Defaults log_input,log_output records session I/O for replay with sudoreplay.

The safe hardening orderInstallyour keyProve keyloginkeep oldsessionopenKeys-onlypolicysshd -t,thenrestartRestrictusersAllowGroupssshusersEnablefail2banInstallyour keyProve keyloginkeep oldsessionopenKeys-onlypolicysshd -t,thenrestartRestrictusersAllowGroupssshusersEnablefail2ban
The safe order of SSH hardening: install and prove a key first, then apply keys-only policy, restrict users, and enable fail2ban.
05

Filesystem security: permissions, flags and audits

4 min read

S03A taught you rwx, the special bits and umask. Hardening turns that knowledge into audits. Start with defaults: a umask of 027 means new files are 640 and new directories 750, so other users get nothing. Set it system-wide through UMASK 027 in /etc/login.defs, which pam_umask applies at login, or in a script under /etc/profile.d/. Check the live value with umask.

1Audit world-writable files

A world-writable file can be changed by any local account, which turns a small compromise into a big one. Find them, staying on the root filesystem with -xdev so you do not wander into proc, sys or network mounts:

bash
sudo find / -xdev -type f -perm -002 -ls
sudo find / -xdev -type d -perm -002 ! -perm -1000 -ls

The first lists world-writable files. The second lists world-writable directories that lack the sticky bit; without sticky, any user can delete or rename anyone else's files there. /tmp itself is world-writable by design and carries sticky (mode 1777), so it will not appear in the second list. For every hit, decide: tighten the mode, add sticky, or move the file to a proper location.

2Audit SUID and SGID programs

A set-user-ID root program runs with root's effective UID no matter who starts it, so every one is a potential privilege-escalation target. List the whole set and save it as a baseline:

bash
sudo find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -ls | tee /root/suid-baseline.txt

-perm -4000 matches setuid, -perm -2000 setgid. Stock Ubuntu shows a small, well-known set: passwd, su, sudo, mount, umount, newgrp, chfn, chsh, gpasswd, ssh-keysign. Anything else deserves a question: who installed it, and does it need the bit? Reduce the set by removing unused packages, or with sudo chmod u-s /path/to/binary for a program you keep but never need privileged. Do not strip passwd or sudo; you would break password changes and administration. Re-run after every upgrade and diff against the baseline: upgrades restore bits and new packages add programs.

3Protect critical files with chattr

Permissions can be changed by root, and a compromised root-level process changes them freely. goes further: with it set, even root cannot modify the file. Try it on a scratch file:

bash
echo "critical setting" > /home/learner/vault.conf
sudo chattr +i /home/learner/vault.conf
lsattr /home/learner/vault.conf
rm /home/learner/vault.conf
sudo rm /home/learner/vault.conf
sudo chattr -i /home/learner/vault.conf

lsattr prints the inode flags as a row of letters; an i in that row means immutable. Both removes fail with "Operation not permitted", even under sudo, until chattr -i clears the flag. The related +a flag makes a file append-only, which suits log files that must never be rewritten. These flags live in the inode, not in the permission bits, and they work on ext2, ext3, ext4, XFS, Btrfs and f2fs. They do not work on tmpfs, proc, sysfs, or the Windows drives mounted in WSL, so test on the board's ext4 root or a VM.

Going deeperWhat immutable really blocks, and who can clear it

The kernel enforces FS_IMMUTABLE_FL on write, truncate, unlink, link, rename, chmod, chown, utime and extended-attribute changes, checked in the VFS and filesystem paths before the operation proceeds. Setting or clearing the flag requires CAP_LINUX_IMMUTABLE, so root can lift it but a compromised daemon without that capability cannot. From C, use ioctl(fd, FS_IOC_GETFLAGS) and FS_IOC_SETFLAGS. Two operational traps: an immutable file under /etc or /usr makes apt or dpkg fail with "Operation not permitted", so never blanket-apply +i across system directories; and an immutable /etc/shadow silently breaks passwd and chage. Protect a short, documented list, and clear it before maintenance.

chattr and package upgrades

Before running apt upgrade on a hardened board, check for immutable files in the paths the upgrade touches: sudo lsattr -R /etc 2>/dev/null | grep -- '-i-'. Clear the flag, upgrade, then re-apply and record it. An upgrade that fails halfway because of one protected file is a self-inflicted outage.

Filesystem hardening loopFind SUIDandwritableReview eachhitFix orremoveProtectwith chattrRe-scan anddiff↻ Repeat from step 1Find SUIDandwritableReview eachhitFix orremoveProtectwith chattrRe-scan anddiff↻ Repeat from step 1
The filesystem hardening loop: find privileged and writable files, review each hit, fix or remove, protect critical files, then re-scan and diff.
06

Updates: CVEs and automatic patching

3 min read

A names a specific flaw so everyone talks about the same bug. MITRE has run the CVE programme since 1999; the National Vulnerability Database adds severity scores. The lifecycle drives your planning: a researcher reports a bug, the vendor fixes it, the fix and CVE are published, maintainers backport it into stable packages, and your machine is safe only once it installs that package. Attackers read the same publications and scan within hours. Your exposure window is the time you run the unpatched version.

On Ubuntu, security fixes arrive through the -security pocket, for example noble-security on 24.04. Check what is pending and apply it:

bash
sudo apt update
apt list --upgradable
sudo apt upgrade

apt list --upgradable shows candidate versions; security updates are the ones you must not defer. sudo apt autoremove clears orphaned dependencies, including old kernels. Read its list before confirming: on a board, keep one known-good kernel as a fallback, because a bad update plus no fallback means a physical rescue.

1Enable unattended-upgrades

Doing this by hand every week fails exactly when you are busy. The unattended-upgrades package installs security updates automatically. Ubuntu ships it; verify it is enabled:

bash
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

The reconfigure step writes /etc/apt/apt.conf.d/20auto-upgrades, which turns on the periodic apt behaviour:

text
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

The policy lives in /etc/apt/apt.conf.d/50unattended-upgrades. Its Unattended-Upgrade::Allowed-Origins list decides what may be installed automatically; the default admits only the security pocket:

text
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};

That default is the right one for a board: security fixes only, no feature upgrades that could change behaviour mid-deployment. Two systemd timers drive the work: apt-daily.timer refreshes lists and downloads, apt-daily-upgrade.timer installs. Inspect them with systemctl list-timers apt-daily*. Before trusting the setup, dry-run it:

bash
sudo unattended-upgrade --dry-run --debug

The dry run prints which packages would be upgraded and why, without changing anything. Results land in /var/log/unattended-upgrades/.

2Plan the reboot, not just the patch

Some fixes, especially kernel and libc updates, take effect only after a restart. The config offers Unattended-Upgrade::Automatic-Reboot "true" with Automatic-Reboot-Time "02:00". On a server that is reasonable; on a sensor node it is a decision: a reboot interrupts capture and adds flash write cycles. Prefer a planned maintenance window, or the A/B image updates with rollback that later embedded modules build. And after apt autoremove, keep one known-good kernel as a recovery path.

BusyBox and minimal images

A Buildroot or BusyBox image has no apt at all. There, patching means rebuilding the image with updated package versions and flashing it, which is why such products pair a read-only root filesystem with a signed update mechanism. The principle is identical: shrink the window between a published fix and a running fix. The mechanics move from the package manager to the build system.

From CVE to running fixCVEpublishedDistroships fixapt-dailydownloadsAuto-install securityReboot ifneededplannedwindow onboardsCVEpublishedDistroships fixapt-dailydownloadsAuto-install securityReboot ifneededplannedwindow onboards
The patch pipeline: a CVE is published, the distribution ships a fix, apt-daily downloads it, unattended-upgrades installs it, and a reboot applies it when needed.
07

Kernel hardening knobs with sysctl

5 min read

The kernel exposes hundreds of tunables under /proc/sys. The sysctl command reads and writes them by dotted name: kernel.randomize_va_space is the file /proc/sys/kernel/randomize_va_space. Read the security-relevant ones first:

bash
sysctl kernel.randomize_va_space kernel.kptr_restrict kernel.dmesg_restrict
sysctl net.ipv4.ip_forward kernel.sysrq

is controlled by kernel.randomize_va_space: 0 disables it, 1 randomises the stack, shared libraries and the vdso, and 2 also randomises the brk heap. The value should be 2, and on Ubuntu it is. ASLR forces an exploit to guess addresses instead of hard-coding them, which turns many memory-corruption attacks from reliable into probabilistic. kernel.kptr_restrict hides kernel pointers from unprivileged readers of proc files when set to 1 or higher; kernel.dmesg_restrict restricts the kernel log buffer to privileged readers when set to 1, which is why Ubuntu asks for sudo dmesg. Both remove information that helps an attacker build a kernel exploit.

net.ipv4.ip_forward decides whether the machine routes packets between interfaces. A sensor node is not a router; the value should be 0 unless you deliberately built a gateway in S07B. An unintended forwarder can bridge networks you meant to keep apart.

kernel.sysrq controls the magic System Request keys, which can force a sync, remount filesystems read-only, or reboot even when userspace is hung. The value is a bitmask: 0 disables all, 1 enables all, and subsets use 16 for sync, 32 for read-only remount and 128 for reboot and power-off. Many installs leave a safe subset enabled. For an unattended board, decide explicitly: 0 removes a local attack avenue, while sync and read-only remount can still save a filesystem during a hang. Read yours, choose, document.

Toggle the bits below and watch the decimal total that you would write into the sysctl file:

kernel.sysrq is a sum of bits, not a menu choice: 48 is 16 plus 32, so only sync and remount read-only are allowed. Set bit 7 to add 128 for a total of 176, which also allows reboot and power-off; 0 disables every function and 1 is the special value that enables all of them.

  • fields: [{"name": "Enable all functions (1)", "lsb": 0, "width": 1}, {"name": "Console logging level (2)", "lsb": 1, "width": 1}, {"name": "Keyboard control, SAK and unraw (4)", "lsb": 2, "width": 1}, {"name": "Process debug dumps (8)", "lsb": 3, "width": 1}, {"name": "Sync (16)", "lsb": 4, "width": 1}, {"name": "Remount read-only (32)", "lsb": 5, "width": 1}, {"name": "Signal processes: term, kill, oom-kill (64)", "lsb": 6, "width": 1}, {"name": "Reboot and power-off (128)", "lsb": 7, "width": 1}]
  • value: 48
  • width: 8

1Make the settings persistent

A runtime write is lost at reboot. This is the single most common sysctl mistake:

bash
sudo sysctl -w kernel.sysrq=0

That changes the running kernel only. Persistence lives in configuration files. Create a drop-in:

text
# /etc/sysctl.d/99-hardening.conf
kernel.randomize_va_space = 2
kernel.kptr_restrict = 1
kernel.dmesg_restrict = 1
net.ipv4.ip_forward = 0
kernel.sysrq = 0

Then load every configured file and verify:

bash
sudo sysctl --system
sysctl kernel.sysrq

sysctl --system reads the standard directories, including /etc/sysctl.d, in order, so your 99-prefixed file applies after the distribution defaults. At boot, systemd-sysctl applies the same files automatically. The next lesson goes deeper into sysctl, sysfs and the rest of the kernel's admin interface.

Going deeperWhat ASLR actually randomises, and its limits

With randomize_va_space=2 the kernel randomises the stack top, the mmap base (which places shared libraries), the vdso, the brk start, and the load address of PIE executables. A non-PIE executable loads at a fixed text address, which is why Ubuntu's gcc builds PIE by default: without PIE, ASLR protects the stack but leaves the code where the attacker expects it. Entropy is smaller on 32-bit architectures, so brute-forcing a 32-bit ARM layout is far more practical than a 64-bit one; prefer 64-bit userspace on capable boards. For debugging, disable ASLR per process with setarch $(uname -m) -R ./program and inspect /proc/PID/maps. ASLR is a mitigation, not a boundary: it raises exploitation cost but does not fix the bug, so patching and confinement stay in the stack.

WSL2 limits

WSL2 shares a Microsoft-built kernel across distributions. Some knobs are read-only, absent, or behave differently, and writes may fail. Treat WSL as a place to read and discuss these settings, and do the write-and-reboot verification in a VM or on the Pi.

The sysctl stackConfig files/etc/sysctl.d/99-hardening.confsystemd-sysctlapplies files at boot/proc/sysruntime view, sysctl -wKernel behaviourASLR, forwarding, sysrqsysctl -w alone is lost at rebootConfig files/etc/sysctl.d/99-hardening.confsystemd-sysctlapplies files at boot/proc/sysruntime view, sysctl -wKernel behaviourASLR, forwarding, sysrqsysctl -w alone is lost atreboot
The sysctl stack: persistent config files are applied at boot into /proc/sys, which is the runtime view of kernel behaviour; sysctl -w writes only the runtime view.
08

Logging, auditing and detection

4 min read

On Ubuntu 24.04 with systemd, the journal is the centre of gravity. sshd, sudo and cron all write to it. Read it by unit, by program, or by priority:

bash
journalctl -u ssh --since today --no-pager
journalctl _COMM=sshd -p warning -b --no-pager

-u ssh follows the SSH unit, _COMM=sshd matches the process name, -p warning filters severity, and -b limits to the current boot. Ubuntu also runs rsyslog, which writes traditional files. Its auth log uses one ISO 8601 timestamp field, so a line looks like 2026-11-16T04:45:42.918293+00:00 labpc sshd[1234]: Failed password for learner from 192.0.2.10 port 51234 ssh2. Older releases used a three-field "Nov 16 04:45:42" prefix; do not write parsers against that old shape on 24.04.

1Hunt brute force in the auth log

You already own the log-analysis tools from S04B. Point them at authentication failures:

bash
sudo grep -E 'Failed password|Invalid user' /var/log/auth.log \
  | grep -oE 'from ([0-9]{1,3}\.){3}[0-9]{1,3}' \
  | sort | uniq -c | sort -rn | head

This extracts the source address from each failure line, counts occurrences, and ranks them. A long tail of addresses with a handful of attempts each is internet background noise. One address with hundreds of attempts is a targeted guesser, and fail2ban should already have banned it. Repeated "Invalid user" lines for names like admin, test or oracle show dictionary scanning. On Raspberry Pi OS Bookworm there is no /var/log/auth.log; the same trail lives in the journal, so run journalctl -u ssh and grep that instead. In WSL without rsyslog, the journal is also your only source.

Check what that pattern really captures before trusting the counts:

The pattern matches the word from followed by four groups of one to three digits separated by dots, so it highlights from 192.0.2.10 on both Failed password lines and from 198.51.100.7 on the Invalid user line, and finds nothing in the sudo line. grep -oE prints exactly those matches, which is what sort, uniq -c and sort -rn then rank by attempt count.

2Add auditd for file and syscall evidence

The journal records what services say about themselves. The kernel audit subsystem records what actually happened, per rules you write, independent of the service's own logging. Install it and add a watch:

bash
sudo apt install auditd
sudo auditctl -w /etc/passwd -p wa -k identity
sudo auditctl -w /etc/shadow -p wa -k identity
sudo auditctl -w /etc/sudoers -p wa -k privilege

-w watches a path, -p wa triggers on writes and attribute changes, and -k tags records with a key you can search by. Runtime rules vanish at reboot, so persist them in /etc/audit/rules.d/identity.rules with the same lines, then load them with sudo augenrules --load. Search and report:

bash
sudo ausearch -k identity -i
sudo aureport --auth

ausearch -i interprets numeric UIDs and syscalls into names; records land in /var/log/audit/audit.log. A watch on passwd, shadow and sudoers answers the question "did anyone change the accounts or the privilege rules?" with kernel-level evidence. Keep rules few and targeted on a board: every rule costs CPU and log writes, and flash wears out. Note that containers and WSL may lack the kernel audit interface; use a VM or the Pi for this part.

3Summarise and watch file integrity

For a daily summary, logwatch parses the logs and mails a report; sudo logwatch --range today --detail high prints one on demand. logcheck mails only security-relevant lines. For tamper detection, AIDE hashes a baseline of critical files and reports changes: initialise the database (on Debian and Ubuntu the aideinit wrapper does this), then run sudo aide --check periodically. After a legitimate change, rebuild with sudo aide --update and promote the new database, or later checks drown in known-good differences. Store the baseline where an attacker cannot rewrite it, ideally off the board. AIDE detects change; it does not prevent it, so it sits behind chattr and permissions, not instead of them.

Where SSH failures landsshdjournaldrsyslog/var/log/auth.loggrep andfail2bansshdjournaldrsyslog/var/log/auth.loggrep andfail2ban
The auth log path on Ubuntu: sshd writes to journald, rsyslog copies to /var/log/auth.log, and grep or fail2ban consume it; the Pi keeps the journal only.
09

Capabilities, confinement and mandatory access control

5 min read

Traditional UNIX privilege is all-or-nothing: effective UID 0 bypasses every check. That is why a bug in any set-user-ID root program is a full compromise. Linux split that power into around forty units. A process can hold just the ones it needs. The classic example is binding a port below 1024, which normally requires root: a custom web server needs CAP_NET_BIND_SERVICE for port 443, not the power to read shadow or load modules.

File capabilities attach units to an executable, stored in the security.capability extended attribute. The tools are setcap and getcap, provided by libcap (the libcap2-bin package on Ubuntu). A real example already on your system is ping, which needs raw sockets:

bash
getcap /usr/bin/ping
cp /usr/bin/ping /home/learner/ping-test
sudo setcap cap_net_raw=+ep /home/learner/ping-test
getcap /home/learner/ping-test
/home/learner/ping-test -c1 127.0.0.1

Typical output of the first command (yours will differ): /usr/bin/ping cap_net_raw=ep. The =ep sets the file's permitted and effective sets: permitted means the process may use the capability, effective means it is active at once. A capability-aware program can take =p instead and raise the capability only when needed, which is tighter. Two traps: a package upgrade replaces the binary and drops its capabilities, and a nosuid mount ignores file capabilities entirely. For services, systemd is the cleaner route: AmbientCapabilities=CAP_NET_BIND_SERVICE grants the unit at start, and CapabilityBoundingSet= caps what the service can ever hold. Inspect a process with grep Cap /proc/PID/status, or getpcap $$ for your shell.

Step by stepWhat exec does to capabilities
  1. Read: the kernel reads the file's security.capability attribute during exec.

  2. Permitted: the file's permitted set is added to the process's new permitted set, masked by the bounding set.

  3. Effective: if the file's effective bit is set, the new permitted capabilities are also made effective at once.

  4. Inheritable: the file's inheritable set is ANDed with the process's inheritable set, adding only what both sides allow.

  5. Run: the program starts with exactly those units, not full root, and a capability-aware program can drop more.

Capabilities shrink what a process may do; confinement shrinks what it may see and reach. A is the kernel mechanism behind containers: the next lesson builds containers from namespaces plus cgroups. goes further and restricts the syscall surface itself; systemd exposes it as SystemCallFilter=, and container runtimes ship default profiles. Together with NoNewPrivileges=true, which blocks gaining privilege through exec, these give a service a small, fixed box to live in.

1Mandatory access control with AppArmor

Permissions and capabilities are still decided by owners and by root; (MAC) is the layer above. SELinux, released by the NSA in 2000 and merged into Linux 2.6.0, labels every file and process and enforces rules over those labels; it is the RHEL and Fedora default, and getenforce reports enforcing, permissive or disabled. AppArmor, mainlined in 2.6.36, confines programs with path-based profiles and is the Ubuntu default. Check what is loaded:

bash
sudo aa-status

You will see profiles in enforce mode and some in complain mode. Enforce blocks violations; complain logs them without blocking, which is how you tune a new profile. To confine your own service, write a profile named after its binary path, start in complain mode, and watch the journal for apparmor="DENIED" lines:

text
# /etc/apparmor.d/usr.local.bin.sensor-collector
#include <tunables/global>
/usr/local/bin/sensor-collector {
  #include <abstractions/base>
  /usr/local/bin/sensor-collector ix,
  /etc/sensor-collector.conf r,
  /var/lib/sensor-collector/** rw,
  /dev/ttyUSB0 rw,
  network inet stream,
}

Load it with sudo apparmor_parser -r /etc/apparmor.d/usr.local.bin.sensor-collector, switch modes with aa-complain and aa-enforce, and re-check aa-status. Each rule grants one access: read a config, read-write a data directory, use the serial device, open TCP streams. Everything else is denied, even if the process runs as root. The later security module builds this out with SELinux, seccomp profiles and secure provisioning.

AppArmor in WSL

The WSL2 kernel is built without AppArmor enabled, so aa-status reports it as disabled even though the userspace tools exist. That is a kernel build choice, not a broken installation. Run the profile exercises in a VM or on the Pi, where Ubuntu enables AppArmor by default.

SELinux versus AppArmorSELinuxAppArmorModellabels onobjectsprofiles perpathModesenforcing,permissiveenforce,complainStatus toolgetenforce,sestatusaa-statusTypical distroRHEL, FedoraUbuntu,Debian
SELinux versus AppArmor: label-based versus path-based confinement, their modes, status tools and typical distributions.
10

Common mistakes and the hardening checklist

4 min read

These are the mistakes that appear in real hardening work, with symptom, cause, diagnosis and fix:

  • Locked out of SSH. Symptom: "Permission denied (publickey)" on every login. Cause: keys-only applied before a working key existed, or the user missing from AllowUsers/AllowGroups. Diagnose from the console or an open session: sudo sshd -T, check authorized_keys content and modes (700 directory, 600 file). Fix from the console, then re-apply key first, policy second.
  • sudo broken for everyone. Symptom: "sudo: parse error in /etc/sudoers" or "no valid sudoers sources found". Cause: a plain-editor typo, or a sudoers.d file with wrong mode or a dot in its name. Recover with pkexec visudo or a root shell; always edit through visudo, which validates on save.
  • apt upgrade fails with "Operation not permitted". Symptom: dpkg cannot overwrite one config file. Cause: a leftover chattr +i. Diagnose with sudo lsattr on the named path; clear the flag, upgrade, re-apply and document.
  • fail2ban banned your own address. Symptom: your management station suddenly cannot connect. Cause: your IP missing from ignoreip. Diagnose from the console with sudo fail2ban-client status sshd; unban with sudo fail2ban-client set sshd unbanip YOUR_IP, then add the address to ignoreip.
  • Port still open after disabling a service. Symptom: ss -tulpn still shows the listener. Cause: socket activation or a second instance. Check systemctl list-units --type=socket and the ss process column; disable the socket too, then re-check.
  • A setting vanished after reboot. Symptom: sysctl values, audit rules or fail2ban tweaks reset. Cause: runtime-only changes. Persist in /etc/sysctl.d, /etc/audit/rules.d and jail.local, then reload and reboot-test.

1Run a baseline audit with Lynis

Checklists keep humans honest; tools keep checklists complete. The CIS Benchmarks are free, consensus-built hardening guides per distribution, and they define what "hardened" means in measurable terms. Lynis is the open-source auditor from CISOfy that checks hundreds of such items:

bash
sudo apt install lynis
sudo lynis audit system

Lynis prints findings with test identifiers and writes /var/log/lynis.log plus /var/log/lynis-report.dat, ending with a hardening index. Treat the score as a thermometer, not a target: a high score bought by disabling logging is a worse system. Read each suggestion, judge it against your threat model, apply it, re-run. On the Pi expect SSH, service and password findings; on WSL expect kernel tests to be skipped, because the kernel is not yours to configure.

2Keep the loop, and add the board items

Hardening is a loop, not a project: baseline audit, fix, re-audit, document, monitor. Re-audit after every image change and package upgrade, and diff your SUID and world-writable baselines at the same time. For embedded products add what server checklists miss: no factory default credentials (Mirai's whole business model), no password SSH, the serial console treated as a trust boundary, a read-only root filesystem where possible, signed updates with rollback, and persistent logging so an incident survives reboot. Each maps to a mechanism you now know.

The hardening loopBaselineauditLynis, CISchecklistFixfindingsRe-auditDocumentMonitorlogs↻ Repeat from step 1BaselineauditLynis, CISchecklistFixfindingsRe-auditDocumentMonitorlogs↻ Repeat from step 1
The hardening loop: baseline audit, fix findings, re-audit, document the changes, and monitor logs, repeating after every system change.
11

Try it

1 min read

The lab pack turns this lesson into three exercises on your Pi: run Lynis, apply a chosen set of fixes, and re-audit to prove the change; audit SUID and world-writable files with find, reduce the set, and demonstrate chattr +i blocking even root; then configure the fail2ban sshd jail and verify a real ban from a second machine. Keep your console access open throughout, and write down every change so you can undo it. When you finish, you should be able to hand someone your hardening notes and have them reproduce the exact state of the board.

Key takeaways

  • Hardening is layers: every listening port, account and writable file is surface you can shrink.
  • Install and prove an SSH key before you disable passwords, or you lock yourself out.
  • passwd -l locks a password, not the account; SSH keys still work until you expire the account.
  • sudoers rules use last-match-wins and shell-style wildcards; grant exact commands, not editors or shells.
  • chattr +i blocks writes, deletes and chmod even for root, until the flag is cleared.
  • unattended-upgrades installs security patches on a timer; a board still needs a planned reboot window.
  • sysctl -w is runtime only; persistence lives in /etc/sysctl.d and is loaded at boot.
  • Capabilities split root into small units; AppArmor and SELinux confine programs even when root runs them.

Reflect

  1. Explain to a colleague, in two sentences: explain attack surface, least privilege and defence in depth, and map a host's surface with ss and systemctl.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close