Why kernel code plays by different rules
4 min readThe previous lesson showed what the kernel is. This one shows how to work on it. The first surprise is that the C you write for the kernel follows different rules from user-space C, even though the compiler is the same gcc or clang. The differences are not style preferences. Each one follows from how the kernel runs.
No C library. A user program links against libc, which supplies printf, malloc, fopen and qsort. Those functions rely on system calls, and the kernel is the thing that implements system calls. The kernel therefore cannot call into libc. It is compiled with -nostdinc, so <stdio.h> is not even on the include path. Instead the kernel carries its own small library: memcpy, strlen, snprintf, sorting helpers and string-to-integer converters such as kstrtoint. Headers come from include/linux/ and are written #include <linux/module.h>. Allocation is kmalloc and kfree, which you met last lesson.
No floating point. On entry to the kernel, the CPU's floating-point and vector registers still hold the user program's values. Saving and restoring them on every system call and interrupt would slow every entry, so the kernel does not touch those registers and does not save them. The build flags forbid the compiler from generating floating-point or SIMD instructions in ordinary kernel code. Code that really needs them, such as an accelerated checksum, wraps the work in kernel_fpu_begin() and kernel_fpu_end() on x86 or kernel_neon_begin() and kernel_neon_end() on arm64, and may not sleep inside. Drivers use integer or fixed-point arithmetic instead. A sensor value in thousandths of a degree is a normal kernel idiom.
A related trap: 64-bit division on 32-bit CPUs. On 32-bit ARM, a / b with 64-bit operands makes the compiler call a helper from libgcc, such as __aeabi_uldivmod. The kernel does not link libgcc, so the build fails with an undefined symbol or the module refuses to load. Use div_u64(), do_div() or the other helpers from <linux/math64.h>.
A small, fixed stack. Each task has a kernel stack of fixed size. It is 8 KB on 32-bit ARM and 32-bit x86, and 16 KB on x86-64 and arm64 with their default configuration. Every function call and every local array consumes it. A char buf[4096]; local is a bug waiting for a deep call chain. Recursion is avoided or strictly bounded, and large buffers are allocated with kmalloc. Modern kernels place stacks in virtually mapped memory with guard pages (CONFIG_VMAP_STACK), so an overflow faults instead of silently corrupting a neighbour. The compiler warns about oversized frames with -Wframe-larger-than.
printk instead of printf. There is no stdout. The kernel has its own logging function, printk, which writes into a ring buffer in kernel memory. You will read it with dmesg. The logging section below covers it fully.
Two more rules come from the design rather than the hardware. First, there is no stable internal API: the interface between the kernel and user space is kept stable for decades, but functions inside the kernel change whenever the maintainers need them to. A driver is expected to live in the kernel tree and be updated with it. Second, a module may call only functions the kernel has exported with EXPORT_SYMBOL or EXPORT_SYMBOL_GPL, and the second form is visible only to modules whose declared license is GPL-compatible.
The source tree and how to read it
2 min readThe kernel source is a single large tree, and its top-level layout is stable enough to learn once. The directories you will use constantly are these.
arch/holds one subdirectory per CPU architecture:x86,arm,arm64,riscvand others. Each contains the boot code, the exception and system-call entry assembly, page-table code and the default configurations underarch/<arch>/configs/. Device trees for ARM boards live inarch/arm/boot/dts/andarch/arm64/boot/dts/.drivers/is by far the largest directory. It is split by device class:drivers/gpio,drivers/i2c,drivers/net,drivers/tty,drivers/usband so on. Almost all the code you write in this module belongs here.kernel/has the core that is not architecture specific: the scheduler (kernel/sched/), signals,fork, timers,printk(kernel/printk/), module loading and tracing underkernel/trace/.mm/is memory management: the buddy allocator, slab,vmalloc, page-fault handling and the page cache plumbing.fs/holds the VFS core and each filesystem:fs/ext4,fs/proc,fs/sysfs.net/is the networking stack: sockets, TCP/IP, netfilter, Bluetooth and Wi-Fi support.include/has the headers:include/linux/for shared kernel headers,include/uapi/for headers that define the user-space ABI, andinclude/asm-generic/.Documentation/is the official documentation, which builds into the kernel.org HTML pages you are citing from.scripts/has build helpers,checkpatch.pl, and the Kconfig tools.tools/has user-space utilities such asperf.init/containsmain.c, which holdsstart_kernel().
Three habits make the tree readable. First, search by symbol, not by file name. git grep -n "module_init" drivers/gpio | head finds drivers quickly, and git grep is faster than grep -r because it searches only tracked files. Second, use make cscope or make tags to build index files for your editor, or use a language server with the compile_commands.json generated by scripts/clang-tools/gen_compile_commands.py. Third, read Documentation/ first for any subsystem: the driver-API documents describe the intended use and are written by the maintainers.
The MAINTAINERS file at the top lists the maintainers, mailing list and files for every subsystem, and scripts/get_maintainer.pl reads it. You will use it when you send a patch.
Run scripts/get_maintainer.pl -f drivers/gpio/gpio-pl061.c to print the maintainers and lists for one file. The same script, given a patch, tells you where to send it.
Kconfig and Kbuild: choosing and building what goes in
4 min readA kernel contains thousands of optional features. the kernel's configuration language and tools, which record each option as a symbol with a value of y, m or n is how you pick them. Each option is a symbol such as CONFIG_GPIO_PL061. A symbol of type bool is either built in (y) or absent (n). A tristate symbol has a third value, m, meaning built as a loadable module. Other types are int, hex and string.
Your choices live in .config at the top of the build directory. You rarely edit it by hand. The common entry points are these.
make defconfig # start from the architecture's default (arch/x86/configs/x86_64_defconfig)
make menuconfig # text menu; needs libncurses-dev, flex and bison
make olddefconfig # keep an existing .config, give every new symbol its default, never ask
make savedefconfig # write a minimal defconfig that reproduces your .config
make tinyconfig # smallest bootable kernel the architecture allows
make localmodconfig # keep only the modules currently loaded (useful for a custom desktop kernel)
make oldconfig is the interactive cousin of olddefconfig: it asks about every new symbol. In scripts and bisects you always want olddefconfig, because nothing may block waiting for input. For an ARM board there is usually a vendor or family defconfig: multi_v7_defconfig for 32-bit ARMv7 boards and plain defconfig for arm64.
Reading a Kconfig entry
Each directory has a Kconfig file defining its symbols. A trimmed, invented example shows the keywords you must recognise.
config DEMO_SENSOR
tristate "Demo temperature sensor"
depends on I2C
select REGMAP_I2C
default n
help
Driver for the demo sensor on an I2C bus.
depends on I2C means the prompt is only shown, and the symbol can only be set, when I2C is enabled. Dependencies also cap the value: a tristate that depends on a symbol set to m can be at most m. select REGMAP_I2C is the opposite: it forces another symbol on whenever this one is on. Be careful. select does not check the selected symbol's own dependencies, so selecting something whose dependencies are unmet produces the warning "unmet direct dependencies detected". The documented advice is to use select only for hidden symbols with no prompt, or for symbols with no dependencies. For a symbol the user can see, use depends on. A softer form, imply, suggests a value that the user may still override.
Kbuild: from the symbol to an object
Each directory also has a Makefile that Kbuild reads. The central idiom connects the two systems.
obj-$(CONFIG_DEMO_SENSOR) += demo_sensor.o
When CONFIG_DEMO_SENSOR is y, this expands to obj-y += demo_sensor.o, and the object is linked into the kernel image. When it is m, the line becomes obj-m += demo_sensor.o, and the object is built as demo_sensor.ko. When it is n the line is obj- += ..., which Kbuild ignores. A driver built from several source files names its parts with demo_sensor-y := core.o i2c.o. Directories are descended into with obj-y += subdir/.
Kbuild generates include/generated/autoconf.h from .config, so C code can test CONFIG_FOO with #ifdef, or IS_ENABLED(CONFIG_FOO), which is true for both y and m.
Going deeperWhy link order matters for built-in code
Objects listed under obj-y are linked into vmlinux in the order Kbuild visits them: directory by directory, following the order of obj-y lines in each Makefile. Initialisation functions registered at the same initcall level run in that link order. That is why a driver that depends on another built-in driver is sometimes moved earlier in a Makefile, and why deferred probe, which you will meet later in this module, exists: it lets a driver retry later instead of depending on link order. Modules have no such constraint, because the module loader resolves dependencies by symbol use.
Building the kernel, natively and cross
3 min readOn Ubuntu 24.04, install the build prerequisites once. A kernel build needs a compiler, make, flex, bison, bc, the OpenSSL and libelf development headers, and ncurses for menuconfig. The dwarves package provides pahole, which is needed only if you enable BTF debug information.
sudo apt install build-essential flex bison bc libssl-dev libelf-dev libncurses-dev dwarves
A native x86-64 build is short.
cd ~/linux
make defconfig
make -j"$(nproc)"
ls -l arch/x86/boot/bzImage vmlinux
The build produces two kinds of kernel file. vmlinux is the uncompressed ELF executable with symbols, and it is the file gdb reads. The bootable file is a compressed image that the boot loader or QEMU loads. Its name depends on the architecture.
For x86 the target is bzImage. The name means "big zImage"; it does not mean bzip2. The compression algorithm is chosen with CONFIG_KERNEL_GZIP, CONFIG_KERNEL_ZSTD and similar options. On arm64 the plain, uncompressed Image is the usual file. On 32-bit ARM the target is zImage plus separate device-tree blobs, built with make dtbs.
1install the cross toolchain
For a 64-bit ARM target on Ubuntu, the package gcc-aarch64-linux-gnu provides compilers named aarch64-linux-gnu-gcc. For 32-bit ARM with hardware floating point in user space, the package is gcc-arm-linux-gnueabihf, with the prefix arm-linux-gnueabihf-.
2set ARCH and CROSS_COMPILE
ARCH selects a directory under arch/. Note that it is arm64 for 64-bit ARM and arm for 32-bit ARM. CROSS_COMPILE is a prefix that is prepended to gcc, ld, objcopy and the rest.
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j"$(nproc)" Image dtbs modules
3install modules into a staging directory
Never run modules_install as root into your own /lib/modules for a cross build. Stage it.
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=$PWD/stage modules_install
This copies each .ko into stage/lib/modules/<release>/ and runs depmod, which writes the dependency and alias maps that modprobe reads. Add INSTALL_MOD_STRIP=1 to strip debug symbols; this reduces size considerably on a board with a small flash. make kernelrelease prints the release string that names that directory.
Exporting the two variables once per shell keeps commands short, but it is easy to forget which shell has them. A common professional habit is a tiny wrapper script or a Makefile that sets them, so that a native build and a cross build can never be mixed in one tree. Use a separate output directory with make O=build-arm64, so that object files for different architectures never share a directory.
Switching ARCH without make mrproper or a separate O= directory leaves stale objects and a .config for the wrong CPU. The symptom is a confusing build error or, worse, an image that does not boot. Use one output directory per target.
Booting your kernel in QEMU with a BusyBox initramfs
5 min readA real board needs flashing, cabling and a serial adapter for every test. QEMU removes all that. It is the fastest feedback loop for kernel work. You need three things: a kernel image, a root filesystem, and a command line. For the root filesystem you do not need a disk. You use an an archive of files that the kernel unpacks into RAM at boot and uses as its first root filesystem, running /init from it.
On Ubuntu, sudo apt install qemu-system-x86 qemu-system-arm busybox-static cpio provides the x86-64 and the ARM and arm64 emulators, a statically linked BusyBox, and the archiver. The package qemu-system-arm contains qemu-system-aarch64. busybox-static installs a single executable with no shared-library dependencies, which is exactly what a bare root filesystem needs.
1lay out the root filesystem
mkdir -p ~/initramfs/{bin,sbin,proc,sys,dev,etc}
cp "$(command -v busybox)" ~/initramfs/bin/busybox
cd ~/initramfs
for a in sh ls cat mount umount insmod rmmod lsmod dmesg uname mkdir echo setsid cttyhack; do
ln -s busybox bin/$a
done
Each name is a relative symbolic link to the one BusyBox binary, which decides what to do from the name it was started under.
2write /init
The kernel runs /init as process 1. In an initramfs it must mount the pseudo-filesystems itself, because nothing has done so yet.
#!/bin/busybox sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs devtmpfs /dev
echo "embeddexai: initramfs up, kernel $(uname -r)"
exec setsid cttyhack sh
Make it executable with chmod +x init. The final line starts a shell attached to the console with job control; cttyhack is a BusyBox applet that does this.
3pack it as a cpio archive
The kernel expects the newc cpio format, usually compressed with gzip.
cd ~/initramfs
find . -print0 | cpio --null -o --format=newc | gzip -9 > ../initramfs.cpio.gz
4boot it
qemu-system-x86_64 -m 512M -nographic \
-kernel ~/linux/arch/x86/boot/bzImage \
-initrd ~/initramfs.cpio.gz \
-append "console=ttyS0 rdinit=/init"
-kernel loads the image directly without a boot loader. -initrd supplies the archive. -append is the kernel command line. -nographic sends the emulated serial port to your terminal, so console=ttyS0 makes the kernel print there. Add -enable-kvm if your host has KVM; without it, QEMU uses software translation, which still boots a small kernel quickly. To quit, press Ctrl-A, then X.
For 64-bit ARM, QEMU's generic virt machine does not model a particular board. It is a clean virtual platform that the kernel supports with the same defconfig.
qemu-system-aarch64 -M virt -cpu cortex-a72 -m 512M -nographic \
-kernel ~/linux/arch/arm64/boot/Image \
-initrd ~/initramfs-arm64.cpio.gz \
-append "console=ttyAMA0"
The virt machine has a PL011 serial port, which Linux names ttyAMA0. QEMU builds a device tree for the machine and passes it to the kernel, which is why no -dtb argument is needed. The arm64 initramfs must hold an arm64 BusyBox, for example one built statically with the cross compiler, not the x86 one from your host.
Edit: change a kernel source file or module source in your editor.
Rebuild: run make; only changed objects are recompiled, so a module takes seconds and a small kernel change far less than a full build.
Repack: if the module changed, copy the .ko into the initramfs tree and rerun the cpio command.
Boot: start QEMU with -kernel, -initrd and -append; the kernel unpacks the archive into RAM.
Run /init: the script mounts /proc, /sys and /dev and drops to a shell.
Test: insmod the module, read dmesg, then quit with Ctrl-A X and repeat.
Wrapping the loop in a Makefile
Typing four commands per test defeats the point. A small Makefile at the project top makes the loop one word.
KDIR ?= $(HOME)/linux
RAMFS = $(HOME)/initramfs
run: initramfs.cpio.gz
qemu-system-x86_64 -m 512M -nographic -kernel $(KDIR)/arch/x86/boot/bzImage \
-initrd $< -append "console=ttyS0 rdinit=/init"
initramfs.cpio.gz: $(RAMFS)/init $(wildcard $(RAMFS)/*.ko)
cd $(RAMFS) && find . -print0 | cpio --null -o --format=newc | gzip -9 > $(CURDIR)/$@
With this, make run repacks only when init or a top-level .ko changed, and boots.
initramfs and initrd
The word the older initial RAM disk: a block-device image that the kernel mounts as the first root through a RAM disk driver still appears in command lines and tools, so the difference matters. An initrd is a filesystem image (for example ext2) loaded into a RAM-backed block device and mounted as root, which is later replaced with pivot_root. An initramfs is a cpio archive unpacked straight into a tmpfs-style rootfs, with no block device and no filesystem driver, and it runs /init. Modern kernels always have an empty rootfs; they unpack your archive into it, and if the image is a cpio archive, it is treated as an initramfs. Desktop distributions use initramfs images to load storage drivers before the real root can be mounted. Embedded systems often use one as the entire system.
Out-of-tree modules and loading them
4 min readA a relocatable object file with a .ko extension that the running kernel can link into itself and later remove is built by Kbuild even when the source lives elsewhere. The trick is that your own Makefile only names the objects, then hands control to the kernel's build system.
obj-m += hello.o
KDIR ?= /lib/modules/$(shell uname -r)/build
all:
$(MAKE) -C $(KDIR) M=$(PWD) modules
clean:
$(MAKE) -C $(KDIR) M=$(PWD) clean
-C $(KDIR) changes into the kernel build directory and reads its top-level Makefile. M=$(PWD) tells that Makefile to come back and build the external directory. KDIR must point at a configured and built tree (or the build symlink of an installed headers package), because the module needs the generated headers and Module.symvers. On Ubuntu the headers package is linux-headers-$(uname -r). For your QEMU kernel, set KDIR to your own build directory: make KDIR=$HOME/linux. For a cross build, add ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- to the command, exactly as for the kernel.
The smallest useful module has only an init function, an exit function and a license tag. The next lesson explains each part; here you only need it to load.
#include <linux/module.h>
#include <linux/init.h>
static int __init hello_init(void)
{
pr_info("hello: loaded\n");
return 0;
}
static void __exit hello_exit(void)
{
pr_info("hello: unloaded\n");
}
module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Smallest loadable module");
Loading, listing and removing
sudo insmod hello.ko # insert one file; no dependency resolution
lsmod # reads /proc/modules and prints name, size, use count, users
modinfo hello.ko # reads the .modinfo section: license, description, parameters
sudo rmmod hello # remove by name; fails if the module is in use
lsmod is a formatter for /proc/modules, whose columns are name, size, reference count, the modules that use it, state, and load address. insmod takes a path and does nothing clever: if the module needs symbols from another module, it fails. modprobe is the smarter tool. It works from a module name, reads modules.dep to load dependencies first, reads modules.alias to map device identifiers to modules, and obeys the option and blacklist lines in /etc/modprobe.d/. Those maps are produced by depmod, which modules_install runs for you. modprobe -r removes a module and any now-unused dependencies.
When hardware appears, the kernel sends a uevent containing a a string that encodes a device's bus and identifiers, matched against alias entries in loadable modules. User-space tools use modprobe to load whichever module claims the alias. The alias entries are generated from the MODULE_DEVICE_TABLE macro in the driver source. That is how a USB stick causes usb-storage to load without anyone typing a command.
Two details trip beginners. Removal can fail: rmmod: ERROR: Module hello is in use means a reference count is above zero, and lsmod shows who holds it. Also, a kernel built without CONFIG_MODULE_UNLOAD refuses rmmod entirely. Finally, a module must match the kernel it was built for. The vermagic string baked into the module records the kernel release and key options; modinfo hello.ko | grep vermagic shows it. A mismatch gives Invalid module format, and dmesg prints the version-magic comparison.
Kernel logging: printk, levels and dynamic debug
4 min readthe kernel's logging function, which appends a message with a severity level to a ring buffer and optionally prints it on the console looks like printf, but it has several differences. It writes to a ring buffer in memory; consoles and dmesg read from that buffer. It can be called from almost any context, including interrupt handlers. The messages need a trailing newline, because the buffer joins fragments that lack one. Each message has one of eight levels.
You rarely write the level by hand. Use the helpers: pr_emerg, pr_alert, pr_crit, pr_err, pr_warn, pr_notice, pr_info and pr_debug. A driver that owns a struct device should prefer dev_err, dev_warn, dev_info and dev_dbg, which prefix each line with the driver and device name. That prefix is what lets you tell which of three identical sensors misbehaved. If you want your own prefix on every pr_* message in one file, define pr_fmt before the includes.
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/module.h>
A few format rules matter. %p prints a hashed pointer value so that addresses are not leaked, %px prints the raw address and should be used only while debugging, and %pS prints a symbol name plus offset for a code address. Printing from a hot path floods the buffer and slows the machine, so use pr_info_ratelimited or dev_err_ratelimited where an error can repeat.
Console level versus buffer
Every message goes into the ring buffer. Whether it also appears on the console depends on the console log level, the first number in /proc/sys/kernel/printk. Messages numerically lower than that level reach the console. The boot parameter loglevel= sets it and quiet lowers it. This is why a message can be in dmesg and yet not appear on a serial console.
Reading the log
sudo dmesg # Ubuntu restricts dmesg to root by default
sudo dmesg --level=err,warn # filter by level
sudo dmesg -w # follow new messages
journalctl -k # kernel messages through systemd-journald
Typical output (yours will differ):
[ 0.000000] Linux version 6.8.0 ...
[ 2.431907] hello: loaded
The number in brackets is seconds since boot. dmesg -T converts it to wall-clock time, but that conversion is approximate after suspend. In a BusyBox initramfs, dmesg is a BusyBox applet and needs no extra setup.
Dynamic debug
pr_debug and dev_dbg compile to nothing unless the file defines DEBUG. When the kernel has CONFIG_DYNAMIC_DEBUG, every such call site becomes a switch you can flip at run time. You need root and a mounted debugfs.
sudo mount -t debugfs none /sys/kernel/debug # if not already mounted
echo 'module hello +p' | sudo tee /sys/kernel/debug/dynamic_debug/control
echo 'file hello.c line 20 +p' | sudo tee /sys/kernel/debug/dynamic_debug/control
The p flag enables printing. Extra flags f, l, m and t add the function name, line number, module name and thread id to each message. You can also enable it at boot with dyndbg="module hello +p" on the command line. Dynamic debug lets you leave diagnostic messages in shipping code with almost no cost when they are off.
Going deeperWhy printk was rewritten
The original printk took a global lock around the log buffer, so calling it from an interrupt or NMI could deadlock or lose messages. Kernel 5.10 replaced the buffer with a lockless ring buffer that supports many writers at once. Console output, which can be slow, stays separate from the buffer, and it remains the hard part. For you, the practical result is that a message is safe to emit from an interrupt handler, while the console may lag behind the buffer.
When it goes wrong: oops, panic, BUG, WARN and taint
5 min readWhen kernel code does something the CPU cannot allow, such as dereferencing an invalid pointer, the CPU raises an exception and the kernel's handler finds that the fault came from kernel mode. The result is an a kernel error report printed after an illegal operation in kernel mode; the kernel usually kills the offending task and tries to continue. By default the kernel prints a register dump and a call trace, kills the current task, marks itself tainted, and keeps running if it can. A held lock or half-updated structure may make later behaviour unreliable, which is why you restart after an oops during development.
A an unrecoverable kernel error after which the system halts or reboots is what happens when the kernel cannot continue: an oops in an interrupt handler, in the idle task or in init, or whenever panic_on_oops is set. The line begins Kernel panic - not syncing:. The boot parameter panic=N reboots after N seconds, which suits unattended embedded devices.
Typical output for a null-pointer dereference in a module on x86-64 (yours will differ):
BUG: kernel NULL pointer dereference, address: 0000000000000008
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 PID: 71 Comm: insmod Tainted: G O 6.8.0 #1
RIP: 0010:broken_init+0x10/0x40 [broken]
Call Trace:
<TASK>
do_one_initcall+0x5a/0x300
do_init_module+0x60/0x230
...
Read it from the top. The first line says what happened and at which address: an address near zero is almost always a null pointer plus a structure offset. [#1] counts oopses since boot. Comm and PID identify the task. Tainted: G O shows the taint letters. RIP (on ARM, PC is at) is the instruction that faulted, shown as function+offset/size [module]. The call trace lists how the kernel got there, innermost first. On 32-bit ARM the header reads Internal error: Oops: ... [#1] SMP ARM.
To turn an offset into a source line, build with debug info (CONFIG_DEBUG_INFO) and use scripts/decode_stacktrace.sh vmlinux < oops.txt, or scripts/faddr2line vmlinux broken_init+0x10/0x40. Both use the vmlinux of the exact kernel that crashed.
BUG, WARN and the taint flags
Kernel code can also report problems on purpose. WARN_ON(cond) and WARN(cond, ...) print a backtrace and carry on, which suits a situation that is wrong but survivable; WARN_ON_ONCE limits the noise. BUG() and BUG_ON(cond) trigger an oops deliberately and are discouraged by the kernel's own documentation, because killing a task is rarely better than recovering. Use WARN_ON_ONCE for an assumption you want to see violated, and return an error code.
The letters and a bitmask recording that the kernel has run in an unsupported or suspect state, kept in /proc/sys/kernel/tainted exist so that bug reports can be triaged. Common ones: P for a proprietary module, F for a module force-loaded, D for a kernel that has died with an oops or BUG, W for a warning issued, O for an out-of-tree module, and E for an unsigned module. Loading your own hello.ko therefore sets O, and also E when the kernel is built with module-signature support and the module is unsigned. Taint is never cleared until reboot, and maintainers may ignore a report from a tainted kernel.
cat /proc/sys/kernel/tainted # a number: the sum of the set bits
The value is a bitmask, so you decode it by bit position. The widget below lets you toggle the bits and see which letters are set.
Style, tooling and kernel Git workflows
3 min readThe kernel's coding style is documented in Documentation/process/coding-style.rst. In short: indent with tabs that are eight columns wide, keep braces on the same line as if and for but on their own line for functions, name things in lowercase with underscores, and avoid typedefs for structures. A line length of 80 columns was traditional; checkpatch.pl now warns beyond 100. You do not need to memorise the rules. You run a script.
scripts/checkpatch.pl --strict -f drivers/demo/demo_sensor.c # check a file
git format-patch -1 && scripts/checkpatch.pl 0001-*.patch # check a patch
make C=1 drivers/demo/ # run sparse on files that are rebuilt
make C=2 drivers/demo/ # run sparse on all files
make coccicheck MODE=report # run Coccinelle semantic patches
checkpatch.pl finds style violations and common mistakes in a patch. sparse, installed as the sparse package, checks types the compiler ignores, such as __user pointers, endianness annotations and locking context. This is the tool that catches a user pointer dereferenced directly. a semantic patching tool that matches and rewrites C code by its structure rather than its text, used in the kernel via scripts under scripts/coccinelle finds patterns such as a missing kfree on an error path or a wrong comparison. Run these three on every change before you send it, and fix every warning you cannot justify.
Git: releases, branches and bisect
Kernel work is a Git workflow. Torvalds' mainline tree is at git.kernel.org, and release tags are named v6.8, v6.12 and so on. Stable and long-term maintenance branches live in the separate stable tree as linux-6.6.y and similar; consult the releases page on kernel.org for which series are currently supported, because that list changes.
git clone --branch v6.8 --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git switch -c mywork # start your own branch
git describe # shows the nearest tag, handy for kernelrelease
A shallow --depth 1 clone is quick but cannot bisect; for history, clone fully, or deepen later with git fetch --unshallow. When something worked on one release and fails on another, git bisect halves the range of commits until it names the first bad one.
git bisect start
git bisect bad v6.8
git bisect good v6.6
# build, boot in QEMU, test; then tell git the result
git bisect good # or: git bisect bad
git bisect reset
Each step needs a rebuild, so automate it. Write a script that runs make olddefconfig, builds, boots QEMU with a timeout, and exits 0 for good, 1 for bad, or 125 to skip a commit that does not build. Then git bisect run ./test.sh runs the whole search unattended. The QEMU loop from earlier makes it practical, because each test takes seconds, not minutes of flashing.
Keep your .config fixed across the bisect. Run make olddefconfig after each checkout so new symbols get defaults without prompting; otherwise make may stop and ask a question, and the script hangs.
Debugging early: earlycon, gdb on QEMU, kgdboc, ftrace and config fragments
5 min readEarly console
The normal serial driver is initialised partway through boot, so a crash before that point prints nothing. An early console writes straight to the UART using the minimal code needed. On arm64 virt, earlycon with no argument is enough, because the kernel reads the console address from the device tree's stdout-path; you can also spell it out as earlycon=pl011,0x09000000. On x86 use earlycon=uart8250,io,0x3f8 or the older earlyprintk=serial,ttyS0,115200. Keep console= as well: earlycon covers the beginning of boot, and console= takes over once the real driver is up. Add ignore_loglevel when you want every message to reach the console no matter its level. On a real board, earlycon needs the UART's physical address and register layout to be correct, and a wrong address is silent.
1freeze QEMU before the first instruction
Build with debug information and without address randomisation: enable CONFIG_DEBUG_INFO (choose a DWARF option in menuconfig), and disable CONFIG_RANDOMIZE_BASE, or add nokaslr to the kernel command line so symbols stay where vmlinux says. Then add -s -S to the QEMU command. -s is shorthand for -gdb tcp::1234, a gdb stub on port 1234, and -S halts the virtual CPU at reset.
2attach gdb
gdb ~/linux/vmlinux # use gdb-multiarch for arm64
(gdb) target remote :1234
(gdb) hbreak start_kernel
(gdb) continue
Use a hardware breakpoint (hbreak) for the earliest code: it does not patch kernel memory, so it works before the kernel has set up its own page tables and under KVM. When the breakpoint hits you are in start_kernel in init/main.c, and bt, list and info registers all work. After make scripts_gdb, the kernel provides Python helpers: lx-dmesg prints the log buffer, lx-lsmod lists modules and lx-ps lists tasks, from inside gdb. For module code, lx-symbols loads each module's symbols at its actual address.
Build: compile the kernel with debug info and KASLR off, so vmlinux matches memory.
Start frozen: run QEMU with -s -S; the virtual CPU stops before executing any kernel code.
Attach: start gdb on vmlinux and run target remote :1234 to connect to the stub.
Set a breakpoint: use hbreak start_kernel, then continue; execution runs until the breakpoint.
Inspect: print the stack, registers and source lines; step into early initialisation.
Detach or quit: finish with continue or quit gdb; QEMU keeps running.
kgdboc: debugging a live kernel over serial
On real hardware without a QEMU stub, the kernel's own debugger, KGDB, talks gdb's protocol through a serial line. Enable CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE and debug info, and set kgdboc=ttyS0,115200 on the command line (or write the device to /sys/module/kgdboc/parameters/kgdboc). Trigger a stop with echo g > /proc/sysrq-trigger, or add kgdbwait to stop early in boot. Then run gdb on the host, connect with target remote /dev/ttyUSB0, and debug. The line must not be used by a login shell at the same time. A kernel stopped by KGDB freezes all CPUs, so it is an interactive tool, not a production one.
The first function-graph trace
ftrace is the kernel's built-in tracer. The function-graph tracer prints each call of a chosen function and what it calls. It needs CONFIG_FUNCTION_GRAPH_TRACER and CONFIG_DYNAMIC_FTRACE, and in a BusyBox initramfs you must mount tracefs first.
mount -t tracefs nodev /sys/kernel/tracing
cd /sys/kernel/tracing
echo vfs_read > set_graph_function
echo function_graph > current_tracer
cat /proc/version > /dev/null
head -20 trace
echo nop > current_tracer
Typical output shape (yours will differ in functions and times):
0) | vfs_read() {
0) | rw_verify_area() {
0) 0.2 us | security_file_permission();
0) 0.9 us | }
...
0) + 12.3 us | }
Braces show nesting, and the times show how long each call took. This is the same call tree you studied by reading source last lesson, now observed live. Always reset with echo nop > current_tracer, because a running tracer slows the system.
Kconfig fragments and dependency resolution
A project that is not a full vendor tree should not store a 6,000-line .config. Instead, keep a short fragment of the symbols you care about and merge it into a base defconfig.
cat > qemu-dev.config <<'EOF'
CONFIG_BLK_DEV_INITRD=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_GDB_SCRIPTS=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_RANDOMIZE_BASE is not set
EOF
make defconfig
scripts/kconfig/merge_config.sh -m .config qemu-dev.config
make olddefconfig
merge_config.sh -m merges the fragment into .config without resolving, and it reports every symbol whose value you overrode. Later lines override earlier ones. The final make olddefconfig is the resolver: it applies depends on, pulls in select targets and prompts for nothing. Always check afterwards that a symbol you asked for is still set, using grep CONFIG_FOO .config. If a dependency is missing the resolver silently drops your line, so a fragment can fail without an error. scripts/config --enable CONFIG_FOO edits one symbol from a script, and make savedefconfig produces the minimal form to keep in version control.
Going deeperWhy a fragment line can vanish
Kconfig computes a symbol's visibility from its depends on expression. A symbol that is not visible cannot be set by the user, and its value falls back to its default. olddefconfig evaluates symbols in dependency order, so a line like CONFIG_FOO=y for a symbol that depends on BAR is discarded when BAR is off. The fix is to add the dependency chain to the fragment. The make menuconfig search (press /) shows a symbol's "Depends on" and "Selected by" lists, which is the quickest way to find the missing link.
Common mistakes and how to diagnose them
3 min readThe kernel boots, then panics at the end. Symptom: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0), or No working init found. Cause: no initramfs was supplied, CONFIG_BLK_DEV_INITRD is off, or the archive has no executable /init. Diagnose: read the messages just above the panic for Unpacking initramfs; check zcat initramfs.cpio.gz | cpio -t for init. Fix: enable the option, pass -initrd, and chmod +x init.
/init exists but will not run. Symptom: Failed to execute /init (error -2), or a shell reports required file not found. Cause: the first line of the script names an interpreter that is absent, or the BusyBox is dynamically linked and the loader and libraries are missing. Diagnose: run file bin/busybox and look for statically linked. Fix: copy a static BusyBox, such as the one from busybox-static.
Nothing appears on the screen. Symptom: QEMU starts and stays silent. Cause: console= names the wrong device, or -nographic is missing. Diagnose: add earlycon and ignore_loglevel. Fix: ttyS0 for x86 serial, ttyAMA0 for arm64 virt.
make fails on a certificate file. Symptom: an error naming debian/canonical-certs.pem after copying the distribution's /boot/config-* as .config. Cause: that config points at a signing key your tree lacks. Fix: scripts/config --disable SYSTEM_TRUSTED_KEYS --disable SYSTEM_REVOCATION_KEYS, then make olddefconfig.
insmod: ERROR: could not insert module: Invalid module format. Cause: the module was built for a different kernel release or architecture. Diagnose: compare modinfo -F vermagic hello.ko with uname -r, and read dmesg. Fix: rebuild against the exact tree you boot, with the same ARCH.
Key was rejected by service on a Secure Boot desktop. Cause: the kernel enforces module signatures. Fix: sign the module with an enrolled key, or test in QEMU instead, which is the point of this lesson.
A config line has no effect. Cause: a missing dependency, as above. Diagnose: grep CONFIG_X .config and the menuconfig search. Fix: add the dependencies.
gdb shows odd lines or <optimized out>. Cause: the kernel is always built with optimisation, so variables vanish and lines move. This is normal. Fix: rely on backtraces, pr_debug and tracing, and inspect variables at function entry where they still live in registers.
Try it
1 min readDo the lab for this session: build a minimal kernel and a BusyBox initramfs, boot them in QEMU, and script the loop with a Makefile so a rebuild and boot takes one command. Then load and unload the small module from this lesson, break it on purpose, and read the oops it prints, including the taint letters. Finally, add -s -S, attach gdb, set a breakpoint in start_kernel, and run your first function-graph trace. Finish with the practice pack, which tests each of these skills without repeating the lab steps.
Key takeaways
- Kernel C has no libc, no floating point, a small fixed stack of 8 KB or 16 KB, and printk instead of printf.
- The source tree is organised by role: arch/ for CPU ports, drivers/ for devices, and kernel/, mm/, fs/ and net/ for the core subsystems.
- Kconfig decides what is built and as what (y, m or n); Kbuild turns obj-y and obj-m lines into vmlinux and .ko files.
- ARCH= picks the architecture and CROSS_COMPILE= picks the toolchain prefix; x86 makes bzImage, arm64 makes Image, 32-bit ARM makes zImage.
- A kernel plus a cpio initramfs with a static BusyBox boots under QEMU in seconds and is the fastest feedback loop for kernel work.
- Out-of-tree modules build with make -C $(KDIR) M=$(PWD) modules against a configured tree and load with insmod or modprobe.
- An oops kills the offending task and taints the kernel; a panic stops the machine; WARN reports and continues.
- QEMU's -s -S options expose a gdb stub, so you can break in start_kernel before any serial driver exists.
Reflect
- Explain to a colleague, in two sentences: list the coding rules that separate kernel C from user-space C and explain the hardware or design reason for each
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Official docsThe Linux kernel documentation
- Official docsKernel Build System (Kbuild)
- Official docsKconfig language
- Official docsLinux kernel coding style
- Official docsTainted kernels
- Official docsDynamic debug
- ArticleBootlin Linux kernel and driver development training slides
Official documentation first. Links checked when the lesson was published.