all
Stage 07

Kernel Compilation

Cross-compile the Linux kernel for embedded ARM targets, understand the Kbuild system, manage build outputs, and optimize rebuild times.

Kconfig & menuconfig Boot Parameters
13 min read
63253 chars

Getting the Kernel Source

{:.gc-basic}

Basic

The Linux kernel source is maintained at kernel.org. There are several branches and trees to know about:

Tree Purpose
linux-stable Stable releases — use this for production
linux-next Integration tree for the next merge window
torvalds/linux Linus’s mainline tree
Vendor BSP trees ST, NXP, TI, Raspberry Pi — board-specific patches 
# Clone the stable kernel tree (shallow clone for speed)
$ git clone --depth=1 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
Cloning into 'linux'...
remote: Enumerating objects: 80753, done.
remote: Counting objects: 100% (80753/80753), done.
remote: Compressing objects: 100% (73421/73421), done.
Receiving objects: 100% (80753/80753), 228.47 MiB | 8.21 MiB/s, done.
Resolving deltas: 100% (5893/5893), done.

# For a full clone with full history (takes longer)
$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

# List available stable release tags
$ git tag -l "v6.1*" | tail -10
v6.1.50
v6.1.51
v6.1.52
v6.1.53
v6.1.54
v6.1.55
v6.1.56
v6.1.57
v6.1.58
v6.1.59

# Check out a specific stable release
$ git checkout v6.1.55

# For Raspberry Pi (their patched kernel)
$ git clone --depth=1 https://github.com/raspberrypi/linux

# Download a tarball (no git history, fastest)
$ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.55.tar.xz
$ tar xf linux-6.1.55.tar.xz
$ cd linux-6.1.55

Kernel Source Directory Layout

Understanding the top-level directory structure is essential:

$ ls linux/
arch/       # Architecture-specific code (ARM, x86, MIPS, RISC-V...)
block/      # Block I/O layer
crypto/     # Cryptographic API
Documentation/  # Kernel documentation
drivers/    # All device drivers (~60% of the kernel source)
fs/         # Filesystem implementations (ext4, btrfs, NFS, FAT...)
include/    # Kernel header files
init/       # Early boot code (start_kernel() is here)
ipc/        # Inter-process communication (pipes, semaphores, shared memory)
kernel/     # Core kernel: scheduler, signals, timers, locking
lib/        # Generic kernel library functions
mm/         # Memory management subsystem
net/        # Network stack (TCP/IP, Netfilter, wireless...)
scripts/    # Build scripts, Kconfig tools
security/   # Security frameworks (SELinux, AppArmor, seccomp)
sound/      # ALSA sound subsystem
tools/      # Userspace tools (perf, bpf, selftests)
usr/        # initramfs/initrd generation
virt/       # Virtualization support (KVM)
Makefile    # Top-level Kbuild Makefile
Kconfig     # Top-level Kconfig file

The arch/ directory contains the critical architecture-specific code including:

  • arch/arm/ — 32-bit ARM
  • arch/arm64/ — AArch64 / ARM 64-bit
  • arch/x86/ — x86 and x86_64
  • arch/<arch>/configs/ — defconfig files for boards
  • arch/<arch>/boot/ — where kernel images are placed after build

Cross-Compilation Variables

Cross-compiling means building code on one architecture (your x86_64 development machine) to run on a different architecture (ARM embedded target). Two environment variables control this:

# ARCH: tells the kernel build system which architecture to build for
# CROSS_COMPILE: the prefix for the cross-compiler toolchain binaries

# ARM 32-bit (hard-float ABI)
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-

# ARM 64-bit (AArch64)
export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-

# Verify your toolchain is installed and working
$ ${CROSS_COMPILE}gcc --version
arm-linux-gnueabihf-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.

# Install toolchains on Ubuntu/Debian
$ sudo apt-get install gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu

# You can also pass ARCH and CROSS_COMPILE on the command line
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig

Kernel Image Formats

Different bootloaders and architectures expect different kernel image formats:

Image Architecture Description
vmlinux All ELF binary — uncompressed, not directly bootable
zImage ARM 32-bit Self-decompressing compressed kernel
uImage ARM 32-bit zImage wrapped with U-Boot header (mkimage)
Image AArch64 Uncompressed kernel image (common for 64-bit ARM)
Image.gz AArch64 Gzip-compressed Image
bzImage x86/x86_64 Compressed kernel for PC BIOS/UEFI boot 
# Build specific image formats
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- uImage LOADADDR=0x80008000
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image

# Build Device Tree Blobs (DTBs) — essential for embedded
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- dtbs

# Find the built images
$ ls arch/arm/boot/
compressed/  dts/  Image  install.sh  Makefile  zImage

$ ls arch/arm/boot/dts/ | grep "vexpress"
vexpress-v2m.dtsi
vexpress-v2p-ca15-tc1.dts
vexpress-v2p-ca9.dts

make -j$(nproc) for Parallel Builds

The kernel build system supports parallel compilation. Always use -j to speed up builds:

# $(nproc) returns the number of logical CPU cores
$ nproc
16

# Full build: defconfig + all targets
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- vexpress_defconfig
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc) zImage dtbs modules

# Typical build time on a modern machine:
# First build (cold): ~5-15 minutes
# Incremental build (one changed file): ~10-30 seconds

# Watch build progress
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc) zImage 2>&1 | tail -20
  CC      arch/arm/boot/compressed/misc.o
  CC      arch/arm/boot/compressed/decompress.o
  CC      arch/arm/boot/compressed/string.o
  SHIPPED arch/arm/boot/compressed/lib1funcs.S
  CC      arch/arm/boot/compressed/lib1funcs.o
  SHIPPED arch/arm/boot/compressed/ashldi3.S
  CC      arch/arm/boot/compressed/ashldi3.o
  SHIPPED arch/arm/boot/compressed/bswapsdi2.S
  CC      arch/arm/boot/compressed/bswapsdi2.o
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
Kernel: arch/arm/boot/zImage is ready

Understanding Build Output Files

After a successful build, several important files are generated:

$ ls -lh vmlinux System.map Module.symvers
-rwxrwxr-x 1 user user 128M Mar 7 10:23 vmlinux
-rw-rw-r-- 1 user user 4.1M Mar 7 10:23 System.map
-rw-rw-r-- 1 user user 2.8M Mar 7 10:23 Module.symvers

# vmlinux: the uncompressed kernel ELF binary
# Used for debugging with gdb, generating compressed images
$ file vmlinux
vmlinux: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

# System.map: symbol table mapping kernel symbols to addresses
$ grep " T start_kernel" System.map
c0c00a28 T start_kernel

# Module.symvers: exported symbol versions (for out-of-tree module builds)
$ head -3 Module.symvers
0x00000000    module_layout    vmlinux    EXPORT_SYMBOL
0x00000000    add_taint        vmlinux    EXPORT_SYMBOL_GPL
0x1b7b38d5    seq_escape       vmlinux    EXPORT_SYMBOL

Building and Installing Modules

{:.gc-mid}

Intermediate

Building Kernel Modules

# Build all kernel modules (after building the kernel image)
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc) modules

# Install modules to a staging directory for your rootfs
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- \
    INSTALL_MOD_PATH=/home/user/rootfs \
    modules_install

# What gets installed:
$ find /home/user/rootfs/lib/modules/ -type f | head -20
/home/user/rootfs/lib/modules/6.1.55/kernel/drivers/net/ethernet/intel/e1000/e1000.ko
/home/user/rootfs/lib/modules/6.1.55/kernel/drivers/usb/storage/usb-storage.ko
/home/user/rootfs/lib/modules/6.1.55/kernel/fs/ext4/ext4.ko
/home/user/rootfs/lib/modules/6.1.55/kernel/net/ipv4/netfilter/ip_tables.ko
/home/user/rootfs/lib/modules/6.1.55/modules.alias
/home/user/rootfs/lib/modules/6.1.55/modules.builtin
/home/user/rootfs/lib/modules/6.1.55/modules.dep
/home/user/rootfs/lib/modules/6.1.55/modules.order
/home/user/rootfs/lib/modules/6.1.55/modules.symbols

# The modules.dep file (created by depmod) tracks module dependencies
$ cat /home/user/rootfs/lib/modules/6.1.55/modules.dep | head -5
kernel/drivers/net/ethernet/intel/e1000/e1000.ko:
kernel/drivers/usb/storage/usb-storage.ko: kernel/drivers/scsi/scsi_mod.ko

Without INSTALL_MOD_PATH, modules_install would install to your host system’s /lib/modules/ — almost certainly not what you want when cross-compiling.

Cleaning the Build Tree

# make clean: removes compiled objects and binaries, but keeps .config
# Use this when you want to recompile everything from scratch
$ make clean
  CLEAN   scripts/basic
  CLEAN   scripts/kconfig
  CLEAN   arch/arm/boot/compressed
  CLEAN   arch/arm/boot/dts
  CLEAN   arch/arm/boot
  CLEAN   vmlinux

# make mrproper: removes .config too — full reset of the source tree
# Use this before starting with a completely different config
$ make mrproper
  CLEAN   scripts/basic
  CLEAN   scripts/kconfig
  CLEAN   include/config
  CLEAN   include/generated
  MRPROPER

# make distclean: mrproper + removes editor backup files, patches, etc.
# Use this before making a distribution tarball
$ make distclean

Rule of thumb:

  • make clean — before rebuilding with the same .config
  • make mrproper — before switching architectures or starting fresh
  • make distclean — before packaging source for distribution

Kernel Versioning

# LOCALVERSION lets you append a custom suffix to the kernel version string
# Set in .config or via environment
$ grep LOCALVERSION .config
CONFIG_LOCALVERSION="-myboard-v1"
CONFIG_LOCALVERSION_AUTO=n

# After building with LOCALVERSION:
$ uname -r
6.1.55-myboard-v1

# KERNELRELEASE is the full version string
$ make kernelrelease
6.1.55-myboard-v1

# Check kernel version on a running target
$ uname -r
6.1.55-myboard-v1

$ cat /proc/version
Linux version 6.1.55-myboard-v1 (user@buildhost) (arm-linux-gnueabihf-gcc (Ubuntu 11.4.0) 11.4.0, GNU ld 2.38) #1 SMP Fri Mar 7 10:00:00 UTC 2026

Out-of-Tree Builds

Out-of-tree builds keep the source directory clean by placing all build artifacts in a separate directory. This is essential when you have one kernel source tree and build for multiple boards.

# Build to an external directory with O=
$ mkdir -p /build/kernel-vexpress
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- O=/build/kernel-vexpress vexpress_defconfig
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- O=/build/kernel-vexpress -j$(nproc) zImage dtbs

# The source tree stays untouched, all output goes to /build/kernel-vexpress
$ ls /build/kernel-vexpress/
arch/  drivers/  fs/  include/  init/  .config  Makefile  vmlinux  System.map

# Build for two different boards from the same source
$ make ARCH=arm O=/build/kernel-rpi3 bcm2835_defconfig
$ make ARCH=arm O=/build/kernel-vexpress vexpress_defconfig

Incremental Builds and Dependency Tracking

The Kbuild system tracks file dependencies automatically. Only files that actually changed (or depend on changed files) are recompiled:

# Modify a single driver file
$ touch drivers/gpio/gpio-pl061.c

# Only that file and its dependents recompile
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc) zImage
  CC      drivers/gpio/gpio-pl061.o
  AR      drivers/gpio/built-in.a
  AR      drivers/built-in.a
  LD      vmlinux
  OBJCOPY arch/arm/boot/Image
  GZIP    arch/arm/boot/compressed/piggy_data
  AS      arch/arm/boot/compressed/piggy.o
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
Kernel: arch/arm/boot/zImage is ready
# Total time: ~8 seconds vs 10 minutes for a full build

Building for Raspberry Pi

# Clone Raspberry Pi kernel
$ git clone --depth=1 https://github.com/raspberrypi/linux
$ cd linux

# Raspberry Pi 3 (32-bit)
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi3_defconfig
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc) zImage modules dtbs

# Raspberry Pi 4 (64-bit)
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j$(nproc) Image modules dtbs

# Install to an SD card (assuming it's mounted)
$ mkdir -p /mnt/boot/overlays
$ cp arch/arm64/boot/Image /mnt/boot/kernel8.img
$ cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb /mnt/boot/
$ cp arch/arm64/boot/dts/overlays/*.dtb* /mnt/boot/overlays/
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
    INSTALL_MOD_PATH=/mnt/rootfs modules_install

Advanced Build System Topics

{:.gc-adv}

Advanced

Kbuild: The Kernel Build System

Kbuild is the custom build system used by the Linux kernel. Every directory in the kernel source has a Makefile that lists what to compile:

# drivers/gpio/Makefile

# obj-y: always compiled into the kernel
obj-y                   += gpiolib.o
obj-y                   += gpiolib-legacy.o
obj-y                   += gpiolib-of.o
obj-y                   += gpiolib-devres.o
obj-y                   += gpiolib-cdev.o
obj-y                   += gpiolib-sysfs.o
obj-y                   += gpiolib-acpi.o

# obj-$(CONFIG_GPIO_SYSFS): compiled based on Kconfig option
# If CONFIG_GPIO_SYSFS=y -> compiled into kernel
# If CONFIG_GPIO_SYSFS=m -> compiled as module
# If CONFIG_GPIO_SYSFS=n -> not compiled
obj-$(CONFIG_GPIO_PL061) += gpio-pl061.o
obj-$(CONFIG_GPIO_SYSFS) += gpiolib-sysfs.o
obj-$(CONFIG_GPIO_CDEV)  += gpiolib-cdev.o

Kbuild also supports multi-file modules:

# A module built from multiple source files
obj-$(CONFIG_MY_COMPLEX_DRIVER) += my-complex.o
my-complex-y := my-core.o my-hw.o my-dma.o
my-complex-$(CONFIG_MY_DRIVER_DEBUGFS) += my-debugfs.o

Generating compile_commands.json

Modern IDEs and clangd (the C language server) need compile_commands.json to understand how files are compiled. The kernel can generate this:

# Generate compile_commands.json (requires Python's compdb or bear)
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- compile_commands.json

# Or use bear during the build (alternative approach)
$ bear -- make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc)

# Verify it was created
$ head -20 compile_commands.json
[
  {
    "command": "arm-linux-gnueabihf-gcc -Wp,-MMD,drivers/gpio/.gpio-pl061.o.d ...",
    "directory": "/home/user/linux",
    "file": "drivers/gpio/gpio-pl061.c"
  },
  ...
]

With compile_commands.json, VS Code + clangd gives full IntelliSense, go-to-definition, and cross-references across the entire kernel source.

Ccache for Faster Rebuilds

Ccache caches compilation results so identical compilations are served from cache instead of recompiling:

# Install ccache
$ sudo apt-get install ccache

# Use ccache with cross-compiler
$ make ARCH=arm CROSS_COMPILE="ccache arm-linux-gnueabihf-" -j$(nproc) zImage

# Check ccache statistics
$ ccache --show-stats
Summary:
  Hits:             8432 / 8541 (98.72 %)
    Direct:         8399
    Preprocessed:      33
  Misses:            109
  Uncached:            0
  Primary storage:
    Hits:           8432 / 8541 (98.72 %)
    Misses:          109
    Cache size (GB): 1.23 / 10.00 (12.32 %)

# First build (cold cache): 10 minutes
# Second identical build (warm cache): 45 seconds

Reproducible Kernel Builds

For production firmware, you want builds to be reproducible — the same source always produces bit-identical output:

# Control variables that affect output
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- \
    KBUILD_BUILD_TIMESTAMP="2026-03-07T10:00:00" \
    KBUILD_BUILD_USER="builder" \
    KBUILD_BUILD_HOST="buildserver" \
    -j$(nproc) zImage

# Disable CONFIG_LOCALVERSION_AUTO (embeds git hash into version string)
$ grep LOCALVERSION .config
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_LOCALVERSION=""

# Verify reproducibility: build twice and compare
$ sha256sum arch/arm/boot/zImage
a3f4b2c1... arch/arm/boot/zImage   # Build 1
a3f4b2c1... arch/arm/boot/zImage   # Build 2 (identical)

Applying Custom Patches

# Apply a single patch with git am
$ git am 0001-fix-gpio-driver-race-condition.patch
Applying: fix: gpio: resolve race condition in interrupt handler

# Apply multiple patches from a directory
$ git am patches/*.patch

# Apply with quilt (when git is not available or patches conflict)
$ quilt import patches/0001-fix-gpio.patch
$ quilt push
Applying patch patches/0001-fix-gpio.patch
patching file drivers/gpio/gpio-pl061.c
Now at patch patches/0001-fix-gpio.patch

# Check currently applied patches
$ quilt applied
patches/0001-fix-gpio.patch

# Refresh a patch after editing
$ quilt refresh

# Unapply all patches
$ quilt pop -a

Building with Clang/LLVM

The kernel supports Clang as an alternative to GCC:

# Full LLVM toolchain build (uses clang, lld, llvm-ar, etc.)
$ make ARCH=arm64 LLVM=1 LLVM_IAS=1 -j$(nproc) Image

# Verify compiler used
$ make ARCH=arm64 LLVM=1 -j1 init/main.o 2>&1 | head -3
  CC      init/main.o
clang --target=aarch64-linux-gnu ...

# Install LLVM cross-compilation toolchain
$ sudo apt-get install clang lld llvm

# Clang advantages: better error messages, faster LTO, address sanitizer support
$ make ARCH=arm64 LLVM=1 CONFIG_LTO_CLANG_THIN=y -j$(nproc) Image

Thin LTO (Link-Time Optimization) with Clang can produce smaller, faster kernels by doing cross-module optimization during linking.

Interview Questions

{:.gc-iq}

Q: What is the difference between zImage, uImage, and Image?

zImage is a self-decompressing kernel image for 32-bit ARM. It contains a small decompressor that unpacks the kernel into RAM and jumps to it. uImage is a zImage wrapped with a U-Boot header added by the mkimage tool — the header contains load address, entry point, CRC checksum, and image type metadata that U-Boot uses to validate and boot the image. Image is the uncompressed kernel binary used on AArch64 (64-bit ARM). AArch64 typically uses hardware-accelerated decompression or relies on the bootloader to handle compressed images differently. bzImage is the x86/x86_64 equivalent of zImage, used for PC boots.

Q: What does make modules_install do and why is INSTALL_MOD_PATH important?

make modules_install copies all compiled .ko (kernel object) files to a directory structure under /lib/modules/<kernel-version>/ and runs depmod to generate module dependency files (modules.dep, modules.alias, etc.) that modprobe needs to automatically load modules and their dependencies. INSTALL_MOD_PATH redirects this installation to a staging directory (e.g., the root of your target rootfs) instead of installing to the host system’s /lib/modules/. Without it, you would corrupt your host system’s module directory with ARM binaries.

Q: What is the difference between make clean and make mrproper?

make clean removes compiled objects, .o files, built kernel images, and temporary files, but preserves the .config file. It is used when you want to force a complete recompile while keeping your current configuration. make mrproper does everything clean does plus removes the .config file, generated headers, and all editor/patch artifacts. It resets the source tree to a pristine state equivalent to a fresh checkout. make distclean extends mrproper to also remove editor backup files.

Q: What is Kbuild and how does it determine what gets compiled?

Kbuild is the kernel’s custom recursive Make-based build system. Each directory has a Makefile that uses obj-y (always compile into kernel), obj-m (compile as module), and obj-$(CONFIG_FOO) (compile based on Kconfig value). Kbuild reads the .config file, which defines all CONFIG_* variables. When CONFIG_FOO=y, obj-$(CONFIG_FOO) += foo.o expands to obj-y += foo.o, compiling it into the kernel. When CONFIG_FOO=m, it becomes obj-m += foo.o. Kbuild recursively descends directories that have any object files to build.

Q: How do you cross-compile the kernel for ARM?

Set two variables: ARCH=arm (or arm64 for 64-bit) to tell Kbuild which architecture’s code to compile, and CROSS_COMPILE=arm-linux-gnueabihf- to specify the toolchain prefix. With this prefix, Kbuild calls arm-linux-gnueabihf-gcc for compilation, arm-linux-gnueabihf-ld for linking, etc. A full command looks like: make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc) zImage dtbs modules. You can also export these as environment variables.

Q: What is System.map used for?

System.map is a symbol table that maps every kernel symbol (functions and variables) to its virtual memory address in the running kernel. It is used by debugging tools: when a kernel oops prints an instruction pointer like PC is at 0xc0b4a3f0, System.map can translate that to gpio_request+0x48/0x120. Tools like ksymoops, addr2line, and the kernel’s own oops handler use System.map for symbolic stack traces. It is also used by /proc/kallsyms (the runtime equivalent). System.map is specific to one exact kernel build — it must match the running kernel.

References

{:.gc-ref}

Previous Kconfig & menuconfig Next Boot Parameters