Module 5 · Lesson 2 of 30

Inside the kernel: rules, fast QEMU dev loop

Kernel coding rules, the source tree, Kconfig and Kbuild, cross-building, a minimal QEMU plus BusyBox initramfs loop, module loading, oopses, and gdb on QEMU.

  • Beginner to advanced
  • 26 min read
  • 45 min lab
What you will need

Ubuntu 24.04 (VM, container or WSL2 without KVM works but slowly); QEMU; a Raspberry Pi optional for the cross-build discussion

By the end you can 7 objectives
  • List the coding rules that separate kernel C from user-space C and explain the hardware or design reason for each
  • Navigate the kernel source tree and locate the code that owns a given symbol, driver or configuration option
  • Configure a kernel with defconfig, menuconfig, olddefconfig and a merged configuration fragment, and explain depends on versus select
  • Build a kernel image for x86-64 and arm64 with ARCH and CROSS_COMPILE and name the image file each architecture produces
  • Construct a static BusyBox initramfs and boot it under QEMU using -kernel, -initrd and -append
  • Write an out-of-tree module Makefile, load and unload the module, and interpret printk levels, dynamic debug and dmesg output
  • Analyse an oops or WARN report, read the taint flags, and attach gdb to QEMU to stop in start_kernel
Read the lesson below; the video will appear here when published.
01

Why kernel code plays by different rules

4 min read

The previous lesson showed what the kernel is. This one shows how to work on it. The first surprise is that the C you write for the kernel follows different rules from user-space C, even though the compiler is the same gcc or clang. The differences are not style preferences. Each one follows from how the kernel runs.

No C library. A user program links against libc, which supplies printf, malloc, fopen and qsort. Those functions rely on system calls, and the kernel is the thing that implements system calls. The kernel therefore cannot call into libc. It is compiled with -nostdinc, so <stdio.h> is not even on the include path. Instead the kernel carries its own small library: memcpy, strlen, snprintf, sorting helpers and string-to-integer converters such as kstrtoint. Headers come from include/linux/ and are written #include <linux/module.h>. Allocation is kmalloc and kfree, which you met last lesson.

No floating point. On entry to the kernel, the CPU's floating-point and vector registers still hold the user program's values. Saving and restoring them on every system call and interrupt would slow every entry, so the kernel does not touch those registers and does not save them. The build flags forbid the compiler from generating floating-point or SIMD instructions in ordinary kernel code. Code that really needs them, such as an accelerated checksum, wraps the work in kernel_fpu_begin() and kernel_fpu_end() on x86 or kernel_neon_begin() and kernel_neon_end() on arm64, and may not sleep inside. Drivers use integer or fixed-point arithmetic instead. A sensor value in thousandths of a degree is a normal kernel idiom.

A related trap: 64-bit division on 32-bit CPUs. On 32-bit ARM, a / b with 64-bit operands makes the compiler call a helper from libgcc, such as __aeabi_uldivmod. The kernel does not link libgcc, so the build fails with an undefined symbol or the module refuses to load. Use div_u64(), do_div() or the other helpers from <linux/math64.h>.

A small, fixed stack. Each task has a kernel stack of fixed size. It is 8 KB on 32-bit ARM and 32-bit x86, and 16 KB on x86-64 and arm64 with their default configuration. Every function call and every local array consumes it. A char buf[4096]; local is a bug waiting for a deep call chain. Recursion is avoided or strictly bounded, and large buffers are allocated with kmalloc. Modern kernels place stacks in virtually mapped memory with guard pages (CONFIG_VMAP_STACK), so an overflow faults instead of silently corrupting a neighbour. The compiler warns about oversized frames with -Wframe-larger-than.

printk instead of printf. There is no stdout. The kernel has its own logging function, printk, which writes into a ring buffer in kernel memory. You will read it with dmesg. The logging section below covers it fully.

User C versus kernel CUser spaceKernelLibrarylibckernel's ownhelpersFloating pointallowedavoidedStackgrows tomegabytes8 KB or 16 KBLoggingprintfprintkA crash killsone processpossibly thesystem
The same C language, with different rules, in user space and in the kernel

Two more rules come from the design rather than the hardware. First, there is no stable internal API: the interface between the kernel and user space is kept stable for decades, but functions inside the kernel change whenever the maintainers need them to. A driver is expected to live in the kernel tree and be updated with it. Second, a module may call only functions the kernel has exported with EXPORT_SYMBOL or EXPORT_SYMBOL_GPL, and the second form is visible only to modules whose declared license is GPL-compatible.

02

The source tree and how to read it

2 min read

The kernel source is a single large tree, and its top-level layout is stable enough to learn once. The directories you will use constantly are these.

  • arch/ holds one subdirectory per CPU architecture: x86, arm, arm64, riscv and others. Each contains the boot code, the exception and system-call entry assembly, page-table code and the default configurations under arch/<arch>/configs/. Device trees for ARM boards live in arch/arm/boot/dts/ and arch/arm64/boot/dts/.
  • drivers/ is by far the largest directory. It is split by device class: drivers/gpio, drivers/i2c, drivers/net, drivers/tty, drivers/usb and so on. Almost all the code you write in this module belongs here.
  • kernel/ has the core that is not architecture specific: the scheduler (kernel/sched/), signals, fork, timers, printk (kernel/printk/), module loading and tracing under kernel/trace/.
  • mm/ is memory management: the buddy allocator, slab, vmalloc, page-fault handling and the page cache plumbing.
  • fs/ holds the VFS core and each filesystem: fs/ext4, fs/proc, fs/sysfs.
  • net/ is the networking stack: sockets, TCP/IP, netfilter, Bluetooth and Wi-Fi support.
  • include/ has the headers: include/linux/ for shared kernel headers, include/uapi/ for headers that define the user-space ABI, and include/asm-generic/.
  • Documentation/ is the official documentation, which builds into the kernel.org HTML pages you are citing from. scripts/ has build helpers, checkpatch.pl, and the Kconfig tools. tools/ has user-space utilities such as perf. init/ contains main.c, which holds start_kernel().
Kernel source layoutlinux/arch/one port per CPUdrivers/device driverskernel/scheduler, printk, signalsmm/memory managementfs/VFS and filesystemsnet/network stackinclude/headerslinux/arch/one port per CPUdrivers/device driverskernel/scheduler, printk, signalsmm/memory managementfs/VFS and filesystemsnet/network stackinclude/headers
Top-level directories of the kernel source and what each one owns

Three habits make the tree readable. First, search by symbol, not by file name. git grep -n "module_init" drivers/gpio | head finds drivers quickly, and git grep is faster than grep -r because it searches only tracked files. Second, use make cscope or make tags to build index files for your editor, or use a language server with the compile_commands.json generated by scripts/clang-tools/gen_compile_commands.py. Third, read Documentation/ first for any subsystem: the driver-API documents describe the intended use and are written by the maintainers.

The MAINTAINERS file at the top lists the maintainers, mailing list and files for every subsystem, and scripts/get_maintainer.pl reads it. You will use it when you send a patch.

Find who owns a file

Run scripts/get_maintainer.pl -f drivers/gpio/gpio-pl061.c to print the maintainers and lists for one file. The same script, given a patch, tells you where to send it.

03

Kconfig and Kbuild: choosing and building what goes in

4 min read

A kernel contains thousands of optional features. is how you pick them. Each option is a symbol such as CONFIG_GPIO_PL061. A symbol of type bool is either built in (y) or absent (n). A tristate symbol has a third value, m, meaning built as a loadable module. Other types are int, hex and string.

Your choices live in .config at the top of the build directory. You rarely edit it by hand. The common entry points are these.

bash
make defconfig        # start from the architecture's default (arch/x86/configs/x86_64_defconfig)
make menuconfig       # text menu; needs libncurses-dev, flex and bison
make olddefconfig     # keep an existing .config, give every new symbol its default, never ask
make savedefconfig    # write a minimal defconfig that reproduces your .config
make tinyconfig       # smallest bootable kernel the architecture allows
make localmodconfig   # keep only the modules currently loaded (useful for a custom desktop kernel)

make oldconfig is the interactive cousin of olddefconfig: it asks about every new symbol. In scripts and bisects you always want olddefconfig, because nothing may block waiting for input. For an ARM board there is usually a vendor or family defconfig: multi_v7_defconfig for 32-bit ARMv7 boards and plain defconfig for arm64.

Reading a Kconfig entry

Each directory has a Kconfig file defining its symbols. A trimmed, invented example shows the keywords you must recognise.

text
config DEMO_SENSOR
    tristate "Demo temperature sensor"
    depends on I2C
    select REGMAP_I2C
    default n
    help
      Driver for the demo sensor on an I2C bus.

depends on I2C means the prompt is only shown, and the symbol can only be set, when I2C is enabled. Dependencies also cap the value: a tristate that depends on a symbol set to m can be at most m. select REGMAP_I2C is the opposite: it forces another symbol on whenever this one is on. Be careful. select does not check the selected symbol's own dependencies, so selecting something whose dependencies are unmet produces the warning "unmet direct dependencies detected". The documented advice is to use select only for hidden symbols with no prompt, or for symbols with no dependencies. For a symbol the user can see, use depends on. A softer form, imply, suggests a value that the user may still override.

Kbuild: from the symbol to an object

Each directory also has a Makefile that Kbuild reads. The central idiom connects the two systems.

makefile
obj-$(CONFIG_DEMO_SENSOR) += demo_sensor.o

When CONFIG_DEMO_SENSOR is y, this expands to obj-y += demo_sensor.o, and the object is linked into the kernel image. When it is m, the line becomes obj-m += demo_sensor.o, and the object is built as demo_sensor.ko. When it is n the line is obj- += ..., which Kbuild ignores. A driver built from several source files names its parts with demo_sensor-y := core.o i2c.o. Directories are descended into with obj-y += subdir/.

From Kconfig to imagesKconfigsymbols andrules.configy, m or nKbuildobj-y andobj-mvmlinuxbuilt-incode.ko filesloadablemodulesKconfigsymbols andrules.configy, m or nKbuildobj-y andobj-mvmlinuxbuilt-incode.ko filesloadablemodules
From Kconfig symbols to .config, then Kbuild objects, vmlinux and modules

Kbuild generates include/generated/autoconf.h from .config, so C code can test CONFIG_FOO with #ifdef, or IS_ENABLED(CONFIG_FOO), which is true for both y and m.

Going deeperWhy link order matters for built-in code

Objects listed under obj-y are linked into vmlinux in the order Kbuild visits them: directory by directory, following the order of obj-y lines in each Makefile. Initialisation functions registered at the same initcall level run in that link order. That is why a driver that depends on another built-in driver is sometimes moved earlier in a Makefile, and why deferred probe, which you will meet later in this module, exists: it lets a driver retry later instead of depending on link order. Modules have no such constraint, because the module loader resolves dependencies by symbol use.

04

Building the kernel, natively and cross

3 min read

On Ubuntu 24.04, install the build prerequisites once. A kernel build needs a compiler, make, flex, bison, bc, the OpenSSL and libelf development headers, and ncurses for menuconfig. The dwarves package provides pahole, which is needed only if you enable BTF debug information.

bash
sudo apt install build-essential flex bison bc libssl-dev libelf-dev libncurses-dev dwarves

A native x86-64 build is short.

bash
cd ~/linux
make defconfig
make -j"$(nproc)"
ls -l arch/x86/boot/bzImage vmlinux

The build produces two kinds of kernel file. vmlinux is the uncompressed ELF executable with symbols, and it is the file gdb reads. The bootable file is a compressed image that the boot loader or QEMU loads. Its name depends on the architecture.

Build targets by architecturex86-64arm6432-bit ARMARCH=x86arm64armBoot imagebzImageImagezImageCROSS_COMPILE=none whennativeaarch64-linux-gnu-arm-linux-gnueabihf-Device treenot neededmake dtbsmake dtbs
Image files, ARCH values and toolchain prefixes for three common targets

For x86 the target is bzImage. The name means "big zImage"; it does not mean bzip2. The compression algorithm is chosen with CONFIG_KERNEL_GZIP, CONFIG_KERNEL_ZSTD and similar options. On arm64 the plain, uncompressed Image is the usual file. On 32-bit ARM the target is zImage plus separate device-tree blobs, built with make dtbs.

1install the cross toolchain

For a 64-bit ARM target on Ubuntu, the package gcc-aarch64-linux-gnu provides compilers named aarch64-linux-gnu-gcc. For 32-bit ARM with hardware floating point in user space, the package is gcc-arm-linux-gnueabihf, with the prefix arm-linux-gnueabihf-.

2set ARCH and CROSS_COMPILE

ARCH selects a directory under arch/. Note that it is arm64 for 64-bit ARM and arm for 32-bit ARM. CROSS_COMPILE is a prefix that is prepended to gcc, ld, objcopy and the rest.

bash
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j"$(nproc)" Image dtbs modules

3install modules into a staging directory

Never run modules_install as root into your own /lib/modules for a cross build. Stage it.

bash
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=$PWD/stage modules_install

This copies each .ko into stage/lib/modules/<release>/ and runs depmod, which writes the dependency and alias maps that modprobe reads. Add INSTALL_MOD_STRIP=1 to strip debug symbols; this reduces size considerably on a board with a small flash. make kernelrelease prints the release string that names that directory.

Exporting the two variables once per shell keeps commands short, but it is easy to forget which shell has them. A common professional habit is a tiny wrapper script or a Makefile that sets them, so that a native build and a cross build can never be mixed in one tree. Use a separate output directory with make O=build-arm64, so that object files for different architectures never share a directory.

Do not mix architectures in one tree

Switching ARCH without make mrproper or a separate O= directory leaves stale objects and a .config for the wrong CPU. The symptom is a confusing build error or, worse, an image that does not boot. Use one output directory per target.

05

Booting your kernel in QEMU with a BusyBox initramfs

5 min read

A real board needs flashing, cabling and a serial adapter for every test. QEMU removes all that. It is the fastest feedback loop for kernel work. You need three things: a kernel image, a root filesystem, and a command line. For the root filesystem you do not need a disk. You use an .

On Ubuntu, sudo apt install qemu-system-x86 qemu-system-arm busybox-static cpio provides the x86-64 and the ARM and arm64 emulators, a statically linked BusyBox, and the archiver. The package qemu-system-arm contains qemu-system-aarch64. busybox-static installs a single executable with no shared-library dependencies, which is exactly what a bare root filesystem needs.

1lay out the root filesystem

bash
mkdir -p ~/initramfs/{bin,sbin,proc,sys,dev,etc}
cp "$(command -v busybox)" ~/initramfs/bin/busybox
cd ~/initramfs
for a in sh ls cat mount umount insmod rmmod lsmod dmesg uname mkdir echo setsid cttyhack; do
  ln -s busybox bin/$a
done

Each name is a relative symbolic link to the one BusyBox binary, which decides what to do from the name it was started under.

2write /init

The kernel runs /init as process 1. In an initramfs it must mount the pseudo-filesystems itself, because nothing has done so yet.

sh
#!/bin/busybox sh
mount -t proc     proc     /proc
mount -t sysfs    sysfs    /sys
mount -t devtmpfs devtmpfs /dev
echo "embeddexai: initramfs up, kernel $(uname -r)"
exec setsid cttyhack sh

Make it executable with chmod +x init. The final line starts a shell attached to the console with job control; cttyhack is a BusyBox applet that does this.

3pack it as a cpio archive

The kernel expects the newc cpio format, usually compressed with gzip.

bash
cd ~/initramfs
find . -print0 | cpio --null -o --format=newc | gzip -9 > ../initramfs.cpio.gz

4boot it

bash
qemu-system-x86_64 -m 512M -nographic \
  -kernel ~/linux/arch/x86/boot/bzImage \
  -initrd ~/initramfs.cpio.gz \
  -append "console=ttyS0 rdinit=/init"

-kernel loads the image directly without a boot loader. -initrd supplies the archive. -append is the kernel command line. -nographic sends the emulated serial port to your terminal, so console=ttyS0 makes the kernel print there. Add -enable-kvm if your host has KVM; without it, QEMU uses software translation, which still boots a small kernel quickly. To quit, press Ctrl-A, then X.

For 64-bit ARM, QEMU's generic virt machine does not model a particular board. It is a clean virtual platform that the kernel supports with the same defconfig.

bash
qemu-system-aarch64 -M virt -cpu cortex-a72 -m 512M -nographic \
  -kernel ~/linux/arch/arm64/boot/Image \
  -initrd ~/initramfs-arm64.cpio.gz \
  -append "console=ttyAMA0"

The virt machine has a PL011 serial port, which Linux names ttyAMA0. QEMU builds a device tree for the machine and passes it to the kernel, which is why no -dtb argument is needed. The arm64 initramfs must hold an arm64 BusyBox, for example one built statically with the cross compiler, not the x86 one from your host.

Booting under QEMUQEMUKernel/initload -kernel and-initrdpass -append commandlineunpack cpio intorootfsrun /init as PID 1mount proc, sys, devshell on serial console1. QEMU → Kernelload -kernel and -initrd2. QEMU → Kernelpass -append command line3. Kernel → Kernelunpack cpio into rootfs4. Kernel → /initrun /init as PID 15. /init → /initmount proc, sys, dev6. /init → QEMUshell on serial console
What happens between QEMU starting and the shell prompt appearing
Step by stepOne turn of the fast development loop
  1. Edit: change a kernel source file or module source in your editor.

  2. Rebuild: run make; only changed objects are recompiled, so a module takes seconds and a small kernel change far less than a full build.

  3. Repack: if the module changed, copy the .ko into the initramfs tree and rerun the cpio command.

  4. Boot: start QEMU with -kernel, -initrd and -append; the kernel unpacks the archive into RAM.

  5. Run /init: the script mounts /proc, /sys and /dev and drops to a shell.

  6. Test: insmod the module, read dmesg, then quit with Ctrl-A X and repeat.

Wrapping the loop in a Makefile

Typing four commands per test defeats the point. A small Makefile at the project top makes the loop one word.

makefile
KDIR ?= $(HOME)/linux
RAMFS = $(HOME)/initramfs

run: initramfs.cpio.gz
    qemu-system-x86_64 -m 512M -nographic -kernel $(KDIR)/arch/x86/boot/bzImage \
      -initrd $< -append "console=ttyS0 rdinit=/init"

initramfs.cpio.gz: $(RAMFS)/init $(wildcard $(RAMFS)/*.ko)
    cd $(RAMFS) && find . -print0 | cpio --null -o --format=newc | gzip -9 > $(CURDIR)/$@

With this, make run repacks only when init or a top-level .ko changed, and boots.

initramfs and initrd

The word still appears in command lines and tools, so the difference matters. An initrd is a filesystem image (for example ext2) loaded into a RAM-backed block device and mounted as root, which is later replaced with pivot_root. An initramfs is a cpio archive unpacked straight into a tmpfs-style rootfs, with no block device and no filesystem driver, and it runs /init. Modern kernels always have an empty rootfs; they unpack your archive into it, and if the image is a cpio archive, it is treated as an initramfs. Desktop distributions use initramfs images to load storage drivers before the real root can be mounted. Embedded systems often use one as the entire system.

06

Out-of-tree modules and loading them

4 min read

A is built by Kbuild even when the source lives elsewhere. The trick is that your own Makefile only names the objects, then hands control to the kernel's build system.

makefile
obj-m += hello.o

KDIR ?= /lib/modules/$(shell uname -r)/build

all:
    $(MAKE) -C $(KDIR) M=$(PWD) modules

clean:
    $(MAKE) -C $(KDIR) M=$(PWD) clean

-C $(KDIR) changes into the kernel build directory and reads its top-level Makefile. M=$(PWD) tells that Makefile to come back and build the external directory. KDIR must point at a configured and built tree (or the build symlink of an installed headers package), because the module needs the generated headers and Module.symvers. On Ubuntu the headers package is linux-headers-$(uname -r). For your QEMU kernel, set KDIR to your own build directory: make KDIR=$HOME/linux. For a cross build, add ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- to the command, exactly as for the kernel.

The smallest useful module has only an init function, an exit function and a license tag. The next lesson explains each part; here you only need it to load.

c
#include <linux/module.h>
#include <linux/init.h>

static int __init hello_init(void)
{
    pr_info("hello: loaded\n");
    return 0;
}

static void __exit hello_exit(void)
{
    pr_info("hello: unloaded\n");
}

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Smallest loadable module");

Loading, listing and removing

bash
sudo insmod hello.ko          # insert one file; no dependency resolution
lsmod                         # reads /proc/modules and prints name, size, use count, users
modinfo hello.ko              # reads the .modinfo section: license, description, parameters
sudo rmmod hello              # remove by name; fails if the module is in use

lsmod is a formatter for /proc/modules, whose columns are name, size, reference count, the modules that use it, state, and load address. insmod takes a path and does nothing clever: if the module needs symbols from another module, it fails. modprobe is the smarter tool. It works from a module name, reads modules.dep to load dependencies first, reads modules.alias to map device identifiers to modules, and obeys the option and blacklist lines in /etc/modprobe.d/. Those maps are produced by depmod, which modules_install runs for you. modprobe -r removes a module and any now-unused dependencies.

When hardware appears, the kernel sends a uevent containing a . User-space tools use modprobe to load whichever module claims the alias. The alias entries are generated from the MODULE_DEVICE_TABLE macro in the driver source. That is how a USB stick causes usb-storage to load without anyone typing a command.

Module life cycleNot loadedLiveIn useTransitionsinsmod, init runsNot loadedLivermmod, exit runsLiveNot loadedreference takenLiveIn usereference droppedIn useLiveNot loadedLiveIn useTransitionsinsmod, init runsNot loadedLivermmod, exit runsLiveNot loadedreference takenLiveIn usereference droppedIn useLive
The states of a module and the commands that move it between them

A module is either not loaded or live in the kernel. insmod moves it to live and runs its init function; rmmod moves it back and runs its exit function, but only when no one is using it.

  • states: [{"id": "unloaded", "label": "Not loaded"}, {"id": "live", "label": "Live in kernel"}]
  • transitions: [{"from": "unloaded", "to": "live", "event": "insmod"}, {"from": "live", "to": "unloaded", "event": "rmmod"}]

Two details trip beginners. Removal can fail: rmmod: ERROR: Module hello is in use means a reference count is above zero, and lsmod shows who holds it. Also, a kernel built without CONFIG_MODULE_UNLOAD refuses rmmod entirely. Finally, a module must match the kernel it was built for. The vermagic string baked into the module records the kernel release and key options; modinfo hello.ko | grep vermagic shows it. A mismatch gives Invalid module format, and dmesg prints the version-magic comparison.

07

Kernel logging: printk, levels and dynamic debug

4 min read

looks like printf, but it has several differences. It writes to a ring buffer in memory; consoles and dmesg read from that buffer. It can be called from almost any context, including interrupt handlers. The messages need a trailing newline, because the buffer joins fragments that lack one. Each message has one of eight levels.

printk levels0KERN_EMERGpr_emerg1KERN_ALERTpr_alert2KERN_CRITpr_crit3KERN_ERRpr_err4KERN_WARNINGpr_warn5KERN_NOTICEpr_notice6KERN_INFOpr_info7KERN_DEBUGpr_debuglower number = more urgent
The eight printk levels and the helper macro for each

You rarely write the level by hand. Use the helpers: pr_emerg, pr_alert, pr_crit, pr_err, pr_warn, pr_notice, pr_info and pr_debug. A driver that owns a struct device should prefer dev_err, dev_warn, dev_info and dev_dbg, which prefix each line with the driver and device name. That prefix is what lets you tell which of three identical sensors misbehaved. If you want your own prefix on every pr_* message in one file, define pr_fmt before the includes.

c
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/module.h>

A few format rules matter. %p prints a hashed pointer value so that addresses are not leaked, %px prints the raw address and should be used only while debugging, and %pS prints a symbol name plus offset for a code address. Printing from a hot path floods the buffer and slows the machine, so use pr_info_ratelimited or dev_err_ratelimited where an error can repeat.

Console level versus buffer

Every message goes into the ring buffer. Whether it also appears on the console depends on the console log level, the first number in /proc/sys/kernel/printk. Messages numerically lower than that level reach the console. The boot parameter loglevel= sets it and quiet lowers it. This is why a message can be in dmesg and yet not appear on a serial console.

Reading the log

bash
sudo dmesg                         # Ubuntu restricts dmesg to root by default
sudo dmesg --level=err,warn        # filter by level
sudo dmesg -w                      # follow new messages
journalctl -k                      # kernel messages through systemd-journald

Typical output (yours will differ):

text
[    0.000000] Linux version 6.8.0 ...
[    2.431907] hello: loaded

The number in brackets is seconds since boot. dmesg -T converts it to wall-clock time, but that conversion is approximate after suspend. In a BusyBox initramfs, dmesg is a BusyBox applet and needs no extra setup.

Dynamic debug

pr_debug and dev_dbg compile to nothing unless the file defines DEBUG. When the kernel has CONFIG_DYNAMIC_DEBUG, every such call site becomes a switch you can flip at run time. You need root and a mounted debugfs.

bash
sudo mount -t debugfs none /sys/kernel/debug     # if not already mounted
echo 'module hello +p' | sudo tee /sys/kernel/debug/dynamic_debug/control
echo 'file hello.c line 20 +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

The p flag enables printing. Extra flags f, l, m and t add the function name, line number, module name and thread id to each message. You can also enable it at boot with dyndbg="module hello +p" on the command line. Dynamic debug lets you leave diagnostic messages in shipping code with almost no cost when they are off.

Going deeperWhy printk was rewritten

The original printk took a global lock around the log buffer, so calling it from an interrupt or NMI could deadlock or lose messages. Kernel 5.10 replaced the buffer with a lockless ring buffer that supports many writers at once. Console output, which can be slow, stays separate from the buffer, and it remains the hard part. For you, the practical result is that a message is safe to emit from an interrupt handler, while the console may lag behind the buffer.

08

When it goes wrong: oops, panic, BUG, WARN and taint

5 min read

When kernel code does something the CPU cannot allow, such as dereferencing an invalid pointer, the CPU raises an exception and the kernel's handler finds that the fault came from kernel mode. The result is an . By default the kernel prints a register dump and a call trace, kills the current task, marks itself tainted, and keeps running if it can. A held lock or half-updated structure may make later behaviour unreliable, which is why you restart after an oops during development.

A is what happens when the kernel cannot continue: an oops in an interrupt handler, in the idle task or in init, or whenever panic_on_oops is set. The line begins Kernel panic - not syncing:. The boot parameter panic=N reboots after N seconds, which suits unattended embedded devices.

Typical output for a null-pointer dereference in a module on x86-64 (yours will differ):

text
BUG: kernel NULL pointer dereference, address: 0000000000000008
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 PID: 71 Comm: insmod Tainted: G           O       6.8.0 #1
RIP: 0010:broken_init+0x10/0x40 [broken]
Call Trace:
 <TASK>
 do_one_initcall+0x5a/0x300
 do_init_module+0x60/0x230
 ...

Read it from the top. The first line says what happened and at which address: an address near zero is almost always a null pointer plus a structure offset. [#1] counts oopses since boot. Comm and PID identify the task. Tainted: G O shows the taint letters. RIP (on ARM, PC is at) is the instruction that faulted, shown as function+offset/size [module]. The call trace lists how the kernel got there, innermost first. On 32-bit ARM the header reads Internal error: Oops: ... [#1] SMP ARM.

To turn an offset into a source line, build with debug info (CONFIG_DEBUG_INFO) and use scripts/decode_stacktrace.sh vmlinux < oops.txt, or scripts/faddr2line vmlinux broken_init+0x10/0x40. Both use the vmlinux of the exact kernel that crashed.

From fault to oops or panicIllegalaccessin kernelmodeCPUexceptionOops reportregisters,call traceTaint D setExit from step 3: interrupt, init or panic_on_oopsKernel panicTask killedsystem maygo onIllegalaccessin kernelmodeCPUexceptionOops reportregisters,call traceTaint D setExit from step 3: interrupt,init or panic_on_oopsKernel panicTask killedsystem maygo on
From an illegal kernel access to oops, taint and either continue or panic

BUG, WARN and the taint flags

Kernel code can also report problems on purpose. WARN_ON(cond) and WARN(cond, ...) print a backtrace and carry on, which suits a situation that is wrong but survivable; WARN_ON_ONCE limits the noise. BUG() and BUG_ON(cond) trigger an oops deliberately and are discouraged by the kernel's own documentation, because killing a task is rarely better than recovering. Use WARN_ON_ONCE for an assumption you want to see violated, and return an error code.

The exist so that bug reports can be triaged. Common ones: P for a proprietary module, F for a module force-loaded, D for a kernel that has died with an oops or BUG, W for a warning issued, O for an out-of-tree module, and E for an unsigned module. Loading your own hello.ko therefore sets O, and also E when the kernel is built with module-signature support and the module is unsigned. Taint is never cleared until reboot, and maintainers may ignore a report from a tainted kernel.

bash
cat /proc/sys/kernel/tainted      # a number: the sum of the set bits

The value is a bitmask, so you decode it by bit position. The widget below lets you toggle the bits and see which letters are set.

The taint value 0x3000 has bits 12 and 13 set: O (out-of-tree module) and E (unsigned module), which is what loading your own unsigned module produces on a kernel with signature support.

  • fields: [{"name": "P proprietary", "lsb": 0, "width": 1}, {"name": "F forced", "lsb": 1, "width": 1}, {"name": "S SMP-unsafe", "lsb": 2, "width": 1}, {"name": "R forced unload", "lsb": 3, "width": 1}, {"name": "M machine check", "lsb": 4, "width": 1}, {"name": "B bad page", "lsb": 5, "width": 1}, {"name": "U user request", "lsb": 6, "width": 1}, {"name": "D died", "lsb": 7, "width": 1}, {"name": "A ACPI override", "lsb": 8, "width": 1}, {"name": "W warning", "lsb": 9, "width": 1}, {"name": "C staging", "lsb": 10, "width": 1}, {"name": "I firmware bug", "lsb": 11, "width": 1}, {"name": "O out-of-tree", "lsb": 12, "width": 1}, {"name": "E unsigned", "lsb": 13, "width": 1}]
  • value: 0x3000
  • width: 16
09

Style, tooling and kernel Git workflows

3 min read

The kernel's coding style is documented in Documentation/process/coding-style.rst. In short: indent with tabs that are eight columns wide, keep braces on the same line as if and for but on their own line for functions, name things in lowercase with underscores, and avoid typedefs for structures. A line length of 80 columns was traditional; checkpatch.pl now warns beyond 100. You do not need to memorise the rules. You run a script.

bash
scripts/checkpatch.pl --strict -f drivers/demo/demo_sensor.c   # check a file
git format-patch -1 && scripts/checkpatch.pl 0001-*.patch      # check a patch
make C=1 drivers/demo/                                          # run sparse on files that are rebuilt
make C=2 drivers/demo/                                          # run sparse on all files
make coccicheck MODE=report                                     # run Coccinelle semantic patches

checkpatch.pl finds style violations and common mistakes in a patch. sparse, installed as the sparse package, checks types the compiler ignores, such as __user pointers, endianness annotations and locking context. This is the tool that catches a user pointer dereferenced directly. finds patterns such as a missing kfree on an error path or a wrong comparison. Run these three on every change before you send it, and fix every warning you cannot justify.

Checks before sending a patchEdit codecheckpatch.plstylemake C=1sparsecoccichecksemanticpatternsBuild andbootQEMU testgit bisectif aregressionEdit codecheckpatch.plstylemake C=1sparsecoccichecksemanticpatternsBuild andbootQEMU testgit bisectif aregression
The check-and-build sequence a kernel change passes before it is sent

Git: releases, branches and bisect

Kernel work is a Git workflow. Torvalds' mainline tree is at git.kernel.org, and release tags are named v6.8, v6.12 and so on. Stable and long-term maintenance branches live in the separate stable tree as linux-6.6.y and similar; consult the releases page on kernel.org for which series are currently supported, because that list changes.

bash
git clone --branch v6.8 --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git switch -c mywork               # start your own branch
git describe                       # shows the nearest tag, handy for kernelrelease

A shallow --depth 1 clone is quick but cannot bisect; for history, clone fully, or deepen later with git fetch --unshallow. When something worked on one release and fails on another, git bisect halves the range of commits until it names the first bad one.

bash
git bisect start
git bisect bad  v6.8
git bisect good v6.6
# build, boot in QEMU, test; then tell git the result
git bisect good     # or: git bisect bad
git bisect reset

Each step needs a rebuild, so automate it. Write a script that runs make olddefconfig, builds, boots QEMU with a timeout, and exits 0 for good, 1 for bad, or 125 to skip a commit that does not build. Then git bisect run ./test.sh runs the whole search unattended. The QEMU loop from earlier makes it practical, because each test takes seconds, not minutes of flashing.

Reproduce the configuration at every step

Keep your .config fixed across the bisect. Run make olddefconfig after each checkout so new symbols get defaults without prompting; otherwise make may stop and ask a question, and the script hangs.

10

Debugging early: earlycon, gdb on QEMU, kgdboc, ftrace and config fragments

5 min read

Early console

The normal serial driver is initialised partway through boot, so a crash before that point prints nothing. An early console writes straight to the UART using the minimal code needed. On arm64 virt, earlycon with no argument is enough, because the kernel reads the console address from the device tree's stdout-path; you can also spell it out as earlycon=pl011,0x09000000. On x86 use earlycon=uart8250,io,0x3f8 or the older earlyprintk=serial,ttyS0,115200. Keep console= as well: earlycon covers the beginning of boot, and console= takes over once the real driver is up. Add ignore_loglevel when you want every message to reach the console no matter its level. On a real board, earlycon needs the UART's physical address and register layout to be correct, and a wrong address is silent.

1freeze QEMU before the first instruction

Build with debug information and without address randomisation: enable CONFIG_DEBUG_INFO (choose a DWARF option in menuconfig), and disable CONFIG_RANDOMIZE_BASE, or add nokaslr to the kernel command line so symbols stay where vmlinux says. Then add -s -S to the QEMU command. -s is shorthand for -gdb tcp::1234, a gdb stub on port 1234, and -S halts the virtual CPU at reset.

2attach gdb

bash
gdb ~/linux/vmlinux          # use gdb-multiarch for arm64
(gdb) target remote :1234
(gdb) hbreak start_kernel
(gdb) continue

Use a hardware breakpoint (hbreak) for the earliest code: it does not patch kernel memory, so it works before the kernel has set up its own page tables and under KVM. When the breakpoint hits you are in start_kernel in init/main.c, and bt, list and info registers all work. After make scripts_gdb, the kernel provides Python helpers: lx-dmesg prints the log buffer, lx-lsmod lists modules and lx-ps lists tasks, from inside gdb. For module code, lx-symbols loads each module's symbols at its actual address.

gdb attached to QEMUgdbQEMU stubGuest kerneltarget remote :1234CPU held by -Shbreak start_kernelcontinuereaches start_kernelstopped, inspect1. gdb → QEMU stubtarget remote :12342. QEMU stub → Guest kernelCPU held by -S3. gdb → QEMU stubhbreak start_kernel4. gdb → QEMU stubcontinue5. Guest kernel → QEMU stubreaches start_kernel6. QEMU stub → gdbstopped, inspect
gdb, the QEMU gdb stub and the halted guest kernel
Step by stepStopping in start_kernel with gdb and QEMU
  1. Build: compile the kernel with debug info and KASLR off, so vmlinux matches memory.

  2. Start frozen: run QEMU with -s -S; the virtual CPU stops before executing any kernel code.

  3. Attach: start gdb on vmlinux and run target remote :1234 to connect to the stub.

  4. Set a breakpoint: use hbreak start_kernel, then continue; execution runs until the breakpoint.

  5. Inspect: print the stack, registers and source lines; step into early initialisation.

  6. Detach or quit: finish with continue or quit gdb; QEMU keeps running.

kgdboc: debugging a live kernel over serial

On real hardware without a QEMU stub, the kernel's own debugger, KGDB, talks gdb's protocol through a serial line. Enable CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE and debug info, and set kgdboc=ttyS0,115200 on the command line (or write the device to /sys/module/kgdboc/parameters/kgdboc). Trigger a stop with echo g > /proc/sysrq-trigger, or add kgdbwait to stop early in boot. Then run gdb on the host, connect with target remote /dev/ttyUSB0, and debug. The line must not be used by a login shell at the same time. A kernel stopped by KGDB freezes all CPUs, so it is an interactive tool, not a production one.

The first function-graph trace

ftrace is the kernel's built-in tracer. The function-graph tracer prints each call of a chosen function and what it calls. It needs CONFIG_FUNCTION_GRAPH_TRACER and CONFIG_DYNAMIC_FTRACE, and in a BusyBox initramfs you must mount tracefs first.

bash
mount -t tracefs nodev /sys/kernel/tracing
cd /sys/kernel/tracing
echo vfs_read > set_graph_function
echo function_graph > current_tracer
cat /proc/version > /dev/null
head -20 trace
echo nop > current_tracer

Typical output shape (yours will differ in functions and times):

text
 0)               |  vfs_read() {
 0)               |    rw_verify_area() {
 0)   0.2 us      |      security_file_permission();
 0)   0.9 us      |    }
 ...
 0) + 12.3 us     |  }

Braces show nesting, and the times show how long each call took. This is the same call tree you studied by reading source last lesson, now observed live. Always reset with echo nop > current_tracer, because a running tracer slows the system.

Kconfig fragments and dependency resolution

A project that is not a full vendor tree should not store a 6,000-line .config. Instead, keep a short fragment of the symbols you care about and merge it into a base defconfig.

bash
cat > qemu-dev.config <<'EOF'
CONFIG_BLK_DEV_INITRD=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_GDB_SCRIPTS=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_RANDOMIZE_BASE is not set
EOF
make defconfig
scripts/kconfig/merge_config.sh -m .config qemu-dev.config
make olddefconfig

merge_config.sh -m merges the fragment into .config without resolving, and it reports every symbol whose value you overrode. Later lines override earlier ones. The final make olddefconfig is the resolver: it applies depends on, pulls in select targets and prompts for nothing. Always check afterwards that a symbol you asked for is still set, using grep CONFIG_FOO .config. If a dependency is missing the resolver silently drops your line, so a fragment can fail without an error. scripts/config --enable CONFIG_FOO edits one symbol from a script, and make savedefconfig produces the minimal form to keep in version control.

Going deeperWhy a fragment line can vanish

Kconfig computes a symbol's visibility from its depends on expression. A symbol that is not visible cannot be set by the user, and its value falls back to its default. olddefconfig evaluates symbols in dependency order, so a line like CONFIG_FOO=y for a symbol that depends on BAR is discarded when BAR is off. The fix is to add the dependency chain to the fragment. The make menuconfig search (press /) shows a symbol's "Depends on" and "Selected by" lists, which is the quickest way to find the missing link.

11

Common mistakes and how to diagnose them

3 min read

The kernel boots, then panics at the end. Symptom: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0), or No working init found. Cause: no initramfs was supplied, CONFIG_BLK_DEV_INITRD is off, or the archive has no executable /init. Diagnose: read the messages just above the panic for Unpacking initramfs; check zcat initramfs.cpio.gz | cpio -t for init. Fix: enable the option, pass -initrd, and chmod +x init.

/init exists but will not run. Symptom: Failed to execute /init (error -2), or a shell reports required file not found. Cause: the first line of the script names an interpreter that is absent, or the BusyBox is dynamically linked and the loader and libraries are missing. Diagnose: run file bin/busybox and look for statically linked. Fix: copy a static BusyBox, such as the one from busybox-static.

Nothing appears on the screen. Symptom: QEMU starts and stays silent. Cause: console= names the wrong device, or -nographic is missing. Diagnose: add earlycon and ignore_loglevel. Fix: ttyS0 for x86 serial, ttyAMA0 for arm64 virt.

make fails on a certificate file. Symptom: an error naming debian/canonical-certs.pem after copying the distribution's /boot/config-* as .config. Cause: that config points at a signing key your tree lacks. Fix: scripts/config --disable SYSTEM_TRUSTED_KEYS --disable SYSTEM_REVOCATION_KEYS, then make olddefconfig.

insmod: ERROR: could not insert module: Invalid module format. Cause: the module was built for a different kernel release or architecture. Diagnose: compare modinfo -F vermagic hello.ko with uname -r, and read dmesg. Fix: rebuild against the exact tree you boot, with the same ARCH.

Key was rejected by service on a Secure Boot desktop. Cause: the kernel enforces module signatures. Fix: sign the module with an enrolled key, or test in QEMU instead, which is the point of this lesson.

A config line has no effect. Cause: a missing dependency, as above. Diagnose: grep CONFIG_X .config and the menuconfig search. Fix: add the dependencies.

gdb shows odd lines or <optimized out>. Cause: the kernel is always built with optimisation, so variables vanish and lines move. This is normal. Fix: rely on backtraces, pr_debug and tracing, and inspect variables at function entry where they still live in registers.

Symptoms and first checksSymptomFirst checkNo init foundroot paniccpio has/initRequired filenot foundinit will notrunBusyBox isstaticSilent QEMUno outputconsole= andearlyconInvalid moduleformatinsmod failsvermagic vsuname -rConfig lineignoredsymbol unsetmissingdepends on
Common failures, the first place to look, and the usual fix
12

Try it

1 min read

Do the lab for this session: build a minimal kernel and a BusyBox initramfs, boot them in QEMU, and script the loop with a Makefile so a rebuild and boot takes one command. Then load and unload the small module from this lesson, break it on purpose, and read the oops it prints, including the taint letters. Finally, add -s -S, attach gdb, set a breakpoint in start_kernel, and run your first function-graph trace. Finish with the practice pack, which tests each of these skills without repeating the lab steps.

Key takeaways

  • Kernel C has no libc, no floating point, a small fixed stack of 8 KB or 16 KB, and printk instead of printf.
  • The source tree is organised by role: arch/ for CPU ports, drivers/ for devices, and kernel/, mm/, fs/ and net/ for the core subsystems.
  • Kconfig decides what is built and as what (y, m or n); Kbuild turns obj-y and obj-m lines into vmlinux and .ko files.
  • ARCH= picks the architecture and CROSS_COMPILE= picks the toolchain prefix; x86 makes bzImage, arm64 makes Image, 32-bit ARM makes zImage.
  • A kernel plus a cpio initramfs with a static BusyBox boots under QEMU in seconds and is the fastest feedback loop for kernel work.
  • Out-of-tree modules build with make -C $(KDIR) M=$(PWD) modules against a configured tree and load with insmod or modprobe.
  • An oops kills the offending task and taints the kernel; a panic stops the machine; WARN reports and continues.
  • QEMU's -s -S options expose a gdb stub, so you can break in start_kernel before any serial driver exists.

Reflect

  1. Explain to a colleague, in two sentences: list the coding rules that separate kernel C from user-space C and explain the hardware or design reason for each
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close