Module 2 · Lesson 1 of 30

Make and gdb: a deep dive

Make rules, variables, parallel builds and header dependencies, then a deep dive into gdb: breakpoints, watchpoints, core files, threads, remote debugging.

  • Beginner to advanced
  • 30 min read
  • 50 min lab
What you will need

gcc, make and gdb on Ubuntu 24.04 (WSL2, VM or container); a Raspberry Pi for the remote-debugging section

By the end you can 7 objectives
  • Explain why separate compilation needs a build system and what guarantee make's timestamp-based graph provides.
  • Write Makefiles with rules, automatic variables, .PHONY and the variable flavors =, :=, ?= and +=, separating CFLAGS, CPPFLAGS and LDFLAGS.
  • Use pattern and static-pattern rules, VPATH and -j, and explain the ordering hazards parallel builds expose.
  • Generate automatic header dependencies with -MMD -MP and -include, and verify an incremental rebuild skips untouched objects.
  • Drive gdb: set breakpoints, step, inspect memory, registers and backtraces, and read a stack trace.
  • Use watchpoints, conditional breakpoints and state changes, and diagnose a crash from a core dump.
  • Debug threads and forked processes, and remote-debug an embedded target with gdbserver using DWARF, build-ids and debuginfod.
Read the lesson below; the video will appear here when published.
01

Why make and gdb matter

4 min read

Two failures wait for every systems programmer. The first is the stale build. You edit one header, run the build, and see the program behave exactly as before, because the build system never noticed the change and quietly reused an old object file. The bug is real, but no compiler ever complained, so you chase a phantom. The second is the crash in the field. A device that has run for weeks segfaults at 3 am on a board with no screen and no keyboard, and someone asks you what happened. Both failures are solved by the same pair of tools: a build system that tracks every dependency, and a debugger that can reconstruct a program's state after the fact.

A automates the tedious and error-prone step of deciding what to recompile. The dominant build system on Linux, and the one used by the kernel and U-Boot, is make. A debugger, a program that controls another program's execution, stopping it, inspecting its memory and registers, and letting you step through it, is the other half. The standard debugger is gdb, the GNU debugger. This lesson goes deep into both: how make decides what to rebuild and how to make that decision correct, and how gdb observes and controls a running or crashed program.

The reason these two belong together at the start of a systems-programming module is that everything that follows depends on them. You will build libraries, write daemons, spawn processes, race threads and talk to hardware, and every one of those activities will at some point produce either a wrong build or a crashing program. make and gdb are the tools you reach for first, before reading the code, before adding print statements, before guessing. They are also where the embedded difference shows up first, because on a target board there is no desktop to click, so the command-line forms in this lesson are the forms you will actually use.

From make to debuginfod1976 makeFeldman,Bell Labs1986 gdbStallman,GNUearly 1990sgdbserverGrossman,remotedebug2010 DWARF4debug infoformat2017 DWARF5fasterlookups2019debuginfoddebug fileson demand1976 makeFeldman,Bell Labs1986 gdbStallman,GNUearly 1990sgdbserverGrossman,remotedebug2010 DWARF4debug infoformat2017 DWARF5fasterlookups2019debuginfoddebug fileson demand
Why make and gdb exist: a stale build and a crashed program, each with its tool

Understanding make and gdb also means understanding how we got here. Both are old, and both have survived because their core idea still has no better replacement.

Stuart Feldman wrote make at Bell Labs in 1976. The problem was the one above: on a machine shared by many people, recompiling a whole system after every edit was expensive, and people kept forgetting to recompile the one file they had changed. Feldman's idea was to describe the build as a graph of dependencies and let a program compare file timestamps, rerunning only what was out of date. Almost everything that builds software today, from the Linux kernel to Yocto and Buildroot, is a descendant of that idea. make was even the original motivation behind tab characters in recipes, a small decision that still trips up beginners fifty years later, and which this lesson explains.

gdb is only ten years younger. Richard Stallman wrote it in 1986 as part of the GNU project, aiming to replace the adb and dbx debuggers with one free tool that worked across machines. Its model, stop a program, inspect its state, single-step it, is unchanged, but it has grown enormously: hardware watchpoints, remote debugging through gdbserver, Python scripting, and reverse execution. gdbserver, the tiny program that runs on a target board and lets a host gdb control it, was written by Stu Grossman in the early 1990s and is now the backbone of embedded debugging. The DWARF format that carries debugging information into your binaries reached version 4 in 2010 and version 5 in 2017, and the debuginfod server that downloads debug information on demand appeared in 2019.

make (1976) and gdb (1986) are the two foundations; DWARF reached version 4 in 2010 and version 5 in 2017, and debuginfod appeared in 2019.

  • events: [{"date": "1976", "label": "make", "detail": "Stuart Feldman writes make at Bell Labs; a dependency graph compared by timestamps."}, {"date": "1986", "label": "gdb", "detail": "Richard Stallman writes gdb for the GNU project, replacing adb and dbx."}, {"date": "2010", "label": "DWARF 4", "detail": "The DWARF debugging-information format reaches version 4."}, {"date": "2017", "label": "DWARF 5", "detail": "DWARF reaches version 5."}, {"date": "2019", "label": "debuginfod", "detail": "The debuginfod server appears, serving debug information and source by build-id over HTTP."}]

The history matters because the errors these tools emit, missing separator, undefined reference, Program received signal SIGSEGV, are consequences of design decisions made decades ago. Knowing the model behind the tool is what lets you read those messages as information instead of noise.

02

Make and gdb on embedded Linux

4 min read

The embedded workflow splits the machine in two. Your program is compiled on a development workstation, a PC running Ubuntu, and the result runs on a target board, a Raspberry Pi or a custom ARM device with a fraction of the memory and no comfortable desktop. Compiling for a different machine than the one you build on is cross-compilation, and it changes where each tool lives, not how it works.

make always runs on the host. It is a build tool, part of the development environment, and the target never needs it. When you cross-compile, the only change to your Makefile is the name of the compiler held in a variable: make CC=aarch64-linux-gnu-gcc produces ARM binaries instead of x86 ones. The rules, automatic variables, pattern rules and dependency tracking in this lesson are identical, which is exactly why the embedded world standardized on make and its descendants. The Linux kernel, U-Boot and BusyBox are all built with make, and the CMake and Ninja systems you will meet later generate the same dependency graph that make consumes by hand.

gdb, by contrast, is split in two. The full debugger runs on the host, where it has your source files, symbols and a terminal. On the target runs gdbserver, a program of only a few hundred kilobytes that starts your program, stops it, reads and writes its memory and registers, and talks to the host gdb over a network socket or a serial line. The host gdb does all the smart work, translating your break and print commands into the low-level reads and writes the remote protocol understands. This split is not a workaround; it is the normal way to debug embedded software, because the alternative, a full gdb on the target, would need the target to store all the debugging information, which is far larger than the program itself.

Resource limits shape every choice here. Debugging information is big, often larger than the code it describes, so you strip it from the binary you ship and keep a matching copy on the host. The target may have no disk to write a core dump to, or only a small flash partition, so you either stream the dump over the network or reconstruct the crash from logs and registers. Serial lines are slow, so gdbserver and gdb compress and batch their traffic. These constraints are why the techniques in the later sections, separate debug files, build-ids, debuginfod and gdbserver over the network, exist, and why they are standard practice on embedded Linux rather than optional extras.

Host and target splitHost: makebuilds the binaryHost: gdbfull debugger, symbolsNetwork or serial linkTarget: gdbserverfew hundred KBTarget: programstripped binarythe host controls the target over the linkHost: makebuilds the binaryHost: gdbfull debugger, symbolsNetwork or serial linkTarget: gdbserverfew hundred KBTarget: programstripped binarythe host controls the targetover the link
make on the host, gdbserver on the target, gdb controlling it over a network link
No desktop, no problem

On a headless board you debug exactly as you build: from the host. The sequence is to build with make CC=aarch64-linux-gnu-gcc, copy the stripped binary to the target, run gdbserver :2345 ./prog there, and connect with gdb prog then target remote board:2345 on the host. Every technique in this lesson works unchanged; only the connection differs.

03

Make rules, targets, prerequisites and recipes

3 min read

The unit of a Makefile is the . It has three parts. The target is the file the rule produces, such as an object file or an executable. The prerequisites are the files the target depends on. The are the commands that build the target from the prerequisites.

makefile
monitor: main.o sensor.o
    gcc -o monitor main.o sensor.o

The target is monitor, the prerequisites are main.o and sensor.o, and the recipe is the single gcc line. make's entire job is timestamp comparison. Before running anything, it checks whether the target exists and, if it does, whether it is older than any prerequisite. It runs the recipe only when the target is missing or when at least one prerequisite is newer. Otherwise it reports make: 'monitor' is up to date and does nothing.

Two details are non-negotiable and cause most beginner errors. First, every recipe line must begin with a literal tab, not spaces. make uses the tab to tell a recipe line apart from any other line in the file, and indenting with spaces produces the cryptic error Makefile:3: *** missing separator. Stop. Second, make does not understand C. It has no idea that monitor depends on a header unless you write that header in the prerequisite list. If a rule says sensor.o: sensor.c and you edit sensor.h, make sees no reason to rebuild. This is the stale-build failure from the opening, and it is fixed in a later section.

s handle the targets that produce no file. A clean target is the classic example.

makefile
.PHONY: clean
clean:
    rm -f monitor *.o

Without .PHONY: clean, if a file named clean happened to exist, make would compare its timestamp, find it newer than nothing, and skip the rule. Declaring it phony tells make to treat clean as always out of date and run the recipe every time.

The s make recipes generic. $@ is the target, $< is the first prerequisite, $^ is all prerequisites with duplicates removed, and $? is the subset of prerequisites newer than the target. In the link rule above, $@ is monitor and $^ is main.o sensor.o, so the recipe can be written once and never mention the names again.

makefile
monitor: main.o sensor.o
    gcc -o $@ $^

$? has a specific use: because it lists only the prerequisites that are newer, a recipe such as ar rcs lib.a $? rebuilds an archive using only the changed members, which matters when an archive is large and only one object changed. Two related variables, $* and $(@D), give the stem matched by a pattern rule and the directory part of the target, and are useful in more advanced rules.

Diagram in production
The four automatic variables and what each expands to in a rule
Step by stepHow make decides whether to rebuild
  1. Read the goal: make starts from the first target in the file, or the one you name on the command line.

  2. Walk down: it recurses into each prerequisite, deciding those targets first, so objects are considered before the executable.

  3. Compare timestamps: for each rule it compares the target's modification time with every prerequisite's.

  4. Run if stale: if the target is missing or any prerequisite is newer, it runs the recipe.

  5. Climb back up: it then revisits the parents, relinking only if an object was actually rebuilt.

Write your first rule and watch make skip the up-to-date build
Recording coming soon. Follow the commands in this lesson.
04

Variables, expansions, pattern rules and parallelism

3 min read

A Makefile that spells out one rule per source file is tedious and, worse, drifts as files are added. Two mechanisms remove the repetition: variables and pattern rules.

Variables hold text and are expanded with $(NAME). The four assignment operators differ in when that expansion happens, and the distinction is a real source of bugs. = is a recursively expanded variable: the right-hand side is stored verbatim and expanded every time the variable is used, so later assignments to other variables are seen. := is a simply expanded variable: the right-hand side is expanded once, immediately, using the values current at that point. ?= assigns only if the variable is not already defined, which lets a Makefile offer a default that the command line can override. += appends text, with a space, to the existing value.

makefile
CFLAGS = -Wall -Wextra          # recursive: expanded at use
OPT   := -O2                    # simple: expanded now
CFLAGS += $(OPT) -g             # append
CC    ?= gcc                    # default, overridable by make CC=...

The practical effect is that a simply expanded variable is immune to surprises from later redefinition, while a recursively expanded one can pick up values defined after it, which is occasionally useful but more often a source of confusion when the same variable is set in two places. A fifth operator, !=, assigns the output of a shell command, the modern replacement for the old $(shell ...) function.

The four flag variables follow a convention, not a rule of the language, and separating them is what keeps a Makefile clean. CPPFLAGS holds preprocessor flags, the -I include paths and -D macro definitions, and is used when compiling both C and C++. CFLAGS holds C compilation flags such as -Wall, -O2 and -g. LDFLAGS holds linker flags such as -L search paths and -static. Libraries named with -l conventionally go in a fourth variable, LDLIBS, so the link line reads $(CC) $(LDFLAGS) objects $(LDLIBS). Because the built-in rules use these names, respecting them means make's implicit machinery and your explicit rules agree.

A collapses a dozen object rules into one.

makefile
%.o: %.c
    $(CC) $(CPPFLAGS) $(CFLAGS) -c $<

The % is a wildcard that matches any non-empty string and means the same thing on both sides of the colon. To build sensor.o, make matches %.o with sensor and looks for sensor.c. The recipe uses $<, the matched source. A static pattern rule is a more specific form that lists the targets explicitly and then the pattern, so the rule applies only to those targets and not to every .o in the tree.

makefile
OBJS := main.o sensor.o stats.o
$(OBJS): %.o: %.c
    $(CC) $(CPPFLAGS) $(CFLAGS) -c $<

Even without writing any rule, make knows how to build an object from a C source, because it ships with built-in implicit rules, rules make applies when no explicit rule matches, such as compiling a .c file into a .o using the CC, CPPFLAGS and CFLAGS variables. Running make -p -f /dev/null prints the whole built-in database. This is why an empty Makefile can still compile sensor.o from sensor.c when asked, and why the flag-variable names above are more than a convention: the implicit rules read them.

When sources live in subdirectories, VPATH and vpath tell make where to look. VPATH = src include sets a colon-separated list of directories make searches for any prerequisite it cannot find in the current directory. The vpath directive is the pattern-specific form: vpath %.c src searches only src for .c files. With VPATH = src and a %.o: %.c rule, make finds src/sensor.c and builds sensor.o from it.

Several targets can share one rule. Writing main.o sensor.o: sensor.h adds sensor.h as a prerequisite of both objects, a compact way to express a shared dependency. GNU make 4.3 and later also supports grouped targets, a b c &: prereq, where one recipe invocation builds all three, which matters when a single command produces several outputs.

Parallel builds come from -j. make -j4 runs up to four recipes at once, and bare make -j runs as many as it can. On a multicore machine this can cut a build time by a large factor, but it is only correct if the dependency graph is complete. Parallel make schedules recipes whose prerequisites are ready, and if two objects secretly depend on the same header but neither rule says so, make may compile them in the wrong order or, worse, link the executable before an object is rebuilt. The fix is the subject of the next section: make the graph match reality, then turn the parallelism up. When a parallel build misbehaves, make -n prints the commands without running them, make -B forces a full rebuild, and make --debug=b traces each rebuild decision, all useful for finding the gap.

Explicit rules versus one pattern ruleExplicitrulesOne patternrulemain.omain.o:main.c%.o: %.csensor.osensor.o:sensor.csame rulestats.ostats.o:stats.csame rule
A pattern rule turning a generic source into an object, and a static pattern rule restricted to three targets
Going deeperOrder-only prerequisites, the jobserver, and separate shells

Three subtleties catch most people. The first is a generated directory. If obj/ does not exist, a rule that writes obj/sensor.o fails. Listing the directory as a normal prerequisite would force a rebuild every time its timestamp changes, so make offers order-only prerequisites, written after a |: obj/sensor.o: sensor.c | obj. The directory is created first if needed, but its timestamp is ignored for the rebuild decision. The second is the jobserver. With -j, make creates a pipe and hands out one token per parallel slot; each recipe takes a token, runs, and returns it. A sub-make, or a program that spawns its own make, inherits the pipe through MAKEFLAGS, which is how nested builds share the limit instead of each launching a full parallel tree. The third is that make runs each recipe line in a separate shell by default, so a cd on one line has no effect on the next; the .ONESHELL directive, or a single logical line joined with backslashes, changes that.

Collapse explicit object rules into one pattern rule and turn on -j
Recording coming soon. Follow the commands in this lesson.
05

gdb essentials: starting, breaking and stepping

3 min read

gdb needs one thing from your build: debugging information. The -g flag tells the compiler to embed, in the object file, a description of the source that maps machine code back to files, lines, variables and types. Without it, gdb sees only raw machine code and addresses. Compile with -g and, for clean debugging, -O0 to disable optimization, since an optimizing compiler may reorder code and eliminate variables that you want to inspect.

bash
gcc -g -O0 -Wall -Wextra -o monitor main.c sensor.c
gdb ./monitor

Inside gdb, run (or r) starts the program with no arguments, and run arg1 arg2 passes arguments. start is run with a temporary breakpoint at main, so the program stops at the first line of main immediately, a useful shortcut. A is set with break (or b).

text
(gdb) break sensor_read          # stop at the function entry
(gdb) break sensor.c:42          # stop at a specific line
(gdb) break *main+4              # stop at an address, main plus 4 bytes
(gdb) info breakpoints           # list them, numbered
(gdb) delete 2                   # remove breakpoint 2

A breakpoint can be made conditional, so it stops only when an expression is true, and condition attaches or changes that condition after the fact.

text
(gdb) break sensor.c:42 if reading < 0
(gdb) condition 3 reading < 0   # add a condition to breakpoint 3
(gdb) condition 3               # clear the condition

Once stopped, you move forward with a small set of commands. continue (or c) runs until the next breakpoint, signal or exit. step (or s) runs one source line, stepping into any function call on that line. next (or n) also runs one line but steps over function calls, treating them as a single step. finish runs until the current function returns, and prints the return value, which is the fastest way out of a function you stepped into by accident. until (or u) runs until a source line greater than the current one is reached, useful for exiting a loop without setting a breakpoint; until 99 runs until line 99.

The distinction between step and next is the one beginners mix up most. If the current line calls read_adc(), step stops at the first line inside read_adc, while next runs the whole call and stops at the following line. Use step when you suspect the callee, next when you trust it, and finish to escape when you stepped in by mistake.

The gdb inspect looprunbreakpointstep / nextprint / x/continue↻ Repeat from step 1runbreakpointstep / nextprint / x/continue↻ Repeat from step 1
The core gdb loop: break, run, step or next, inspect, continue
Step by stepWhat a software breakpoint actually does
  1. Insert: gdb writes a trap instruction, on x86-64 the single-byte int3, over the first byte of the target instruction, saving the original byte.

  2. Run: the CPU executes normally until it reaches the overwritten byte.

  3. Trap: the trap instruction raises a SIGTRAP, and the kernel stops the process and notifies gdb.

  4. Report: gdb reports the stop and shows you the source line it mapped from the program counter.

  5. Restore: when you continue, gdb puts the original byte back, single-steps one instruction, and re-inserts the trap.

A breakpoint on a line with no code, or inside code the optimizer moved, will not land where you expect. gdb may report the nearest addressable line instead. This is the first hint that debugging optimized code is harder, a theme the inspection section returns to.

Break, run, step, next and finish on a real crash
Recording coming soon. Follow the commands in this lesson.
06

Automatic dependencies and incremental correctness

3 min read

The Makefile from the last two sections has one weakness: it tracks headers by hand. If sensor.c includes sensor.h and config.h, and the rule lists only sensor.c, then editing config.h changes nothing. make reports 'monitor' is up to date, the binary stays stale, and the bug is invisible. The rule of thumb is that a build that is only correct after make clean has a dependency bug, and make clean before every build is a workaround that masks it.

The compiler knows the real dependency list, and it can write it down. The flags -MMD and -MP make gcc emit, alongside each object, a .d file containing a make rule that lists the object's true prerequisites: its source and every header it included, transitively.

bash
gcc -MMD -MP -c sensor.c   # produces sensor.o and sensor.d

sensor.d contains a line shaped like a make rule, such as sensor.o: sensor.c sensor.h config.h. The -MP flag adds an empty phony target for each header, so deleting a header later does not make make fail on a missing prerequisite. To use the files, the Makefile pulls them in with -include, where the leading dash means ignore the file if it does not exist yet, which is exactly the state on a clean build before any object has been compiled.

makefile
CC      = gcc
CPPFLAGS = -Iinclude
CFLAGS  = -Wall -Wextra -O2 -g -MMD -MP
OBJS    = main.o sensor.o stats.o

monitor: $(OBJS)
    $(CC) -o $@ $^ $(LDFLAGS) $(LDLIBS)

%.o: %.c
    $(CC) $(CPPFLAGS) $(CFLAGS) -c $<

-include $(OBJS:.o=.d)

.PHONY: clean
clean:
    rm -f monitor $(OBJS) $(OBJS:.o=.d)

The expression $(OBJS:.o=.d) rewrites main.o sensor.o stats.o into main.d sensor.d stats.d, and -include reads them all. Now an edit to any header that sensor.c includes, however indirect, appears in sensor.d, and make rebuilds sensor.o and relinks. The hand-maintained header lists are gone, and the dependency graph is generated by the one tool that actually knows it.

-MMD deliberately omits system headers, the ones in /usr/include and your -I paths do not count as system headers, so your own headers are tracked, while the compiler's own headers are not. This is what you want: you do not want to rebuild every object because a system header was touched. The related -MD includes system headers too, which is almost never desirable for application builds and produces enormous dependency files.

Incremental correctness is what makes -j safe. A parallel build can only schedule recipes whose prerequisites are complete, and with the .d files in place, make knows the true prerequisites of every object, so make -j8 neither races nor links prematurely. The performance win is real: on a project with dozens of files, -j8 can turn a minute-long build into a few seconds, but only after the graph is correct. Turn the parallelism up only after the dependency tracking is in place, not before.

Header dependency graphmonitorsensor.osensor.csensor.hconfig.hmain.omain.cconfig.hmonitorsensor.osensor.csensor.hconfig.hmain.omain.cconfig.h
With -MMD, each .d file records the source and every header, so make rebuilds on any header edit
Going deeperHow the .d files are generated on the same pass

-MMD does not run the compiler twice. It tells the preprocessor, which already reads every header to expand includes, to note each one it opens and write the resulting rule to the .d file at the same time it produces the object. The cost is negligible because the preprocessor walks those files anyway. The -MM variant prints the same rule to standard output instead of a file, useful for inspecting what would be tracked, and -M does the same including system headers. For a named output file there is also -MF to choose the .d path and -MT to override the target the rule names, both needed only in unusual layouts.

Prove an incremental rebuild skips the untouched object
Recording coming soon. Follow the commands in this lesson.
07

Inspecting state and reading backtraces

4 min read

Stopping is only useful if you can read what is frozen. gdb offers a hierarchy of inspection, from a named variable to raw bytes to the call stack itself.

print (or p) evaluates an expression in the stopped program's context and prints its value. It understands C expressions, structure members, pointer dereferences and arrays, and takes a format prefix to control the representation.

text
(gdb) print reading            # the value of reading
(gdb) print/x reading          # hexadecimal
(gdb) print sensor->state      # a structure member
(gdb) print arr[0]@5           # the first five elements of arr
(gdb) print ptr                # prints the address in ptr

42 decimal is 0x2a in hex, 0o52 in octal and 0b101010 in binary; this is exactly what print/x shows for a value of 42.

  • value: 42
  • width: 32

display is print that repeats: the expression is evaluated and shown every time the program stops, until you remove it with undisplay. It is the right tool for watching a value change as you step through a loop.

Raw memory inspection is x/, the examine command, with a format, size and count: x/nfu address. n is the number of units, f the format (x hex, d decimal, s string, i instruction), and u the unit size (b byte, h halfword, w word, g giant word). It is how you look at memory that has no variable name, or verify what a pointer really points at.

text
(gdb) x/4bx &reading        # four bytes, hex, at the address of reading
(gdb) x/2gx $rsp            # two 8-byte words at the top of the stack
(gdb) x/s ptr               # the string at ptr, up to the null byte
(gdb) x/8i $pc              # eight instructions starting at the program counter

Register state comes from info registers (or i r), which prints the general-purpose registers, and info registers rax for one. info locals lists the local variables of the current function, and info args its arguments. A lets you save a value for later. Every print result is automatically stored as $1, $2 and so on, and you can assign your own with set $count = reading.

text
(gdb) print reading
$1 = 42
(gdb) set $start = $1       # save it under a readable name
(gdb) print $start * 2
$2 = 84

A , printed with bt, is how you answer the question "how did we get here". Each line is a frame, numbered from zero at the innermost.

text
(gdb) bt
#0  read_adc () at sensor.c:19
#1  0x00005555555551e9 in sensor_read () at sensor.c:31
#2  0x0000555555555254 in main () at main.c:14

This is typical output, and the addresses will differ on your machine. bt full adds the local variables of every frame. frame 2 selects frame two, up and down move between frames, and info frame describes the current frame in detail, including the saved instruction pointer and the frame pointer. When you select an outer frame, print and info locals evaluate in that frame's context, so you can read main's variables while stopped inside read_adc.

Reading the stack itself is how you understand frames from first principles. On x86-64, each call pushes a return address and, if frame pointers are enabled, the old base pointer, and the frame pointer register (rbp) chains the frames together. info registers rbp rsp shows the current base and stack pointers, and x/4gx $rbp dumps the frame. On ARM the same role belongs to the frame pointer register r11, or x29 in the 64-bit ABI.

The call stackhighmain framesaved rbp, return addrsensor_read framelocals, return addrlowread_adc framersp points herehighmain framesaved rbp, return addrsensor_read framelocals, return addrlowread_adc framersp points here
The stack growing downward, with each frame holding a saved base pointer, locals and a return address
Going deeperOptimized code and frame pointers

At -O2 on x86-64, gcc omits the frame pointer and uses rbp as an ordinary register, because a frame pointer costs a register and instructions. gdb still produces a correct backtrace, because the compiler emits unwind information in a DWARF section (.eh_frame and .debug_frame) that tells gdb how to recover the stack layout at every instruction. On ARM and in kernels that need reliable backtraces, -fno-omit-frame-pointer keeps the frame pointer. The practical lesson: you do not need to read raw frames to use bt, but you do need -g, and under optimization some variables are held in registers or eliminated, so print can report <optimized out>. Recompile that one file at -O0 -g when a variable you need disappears.

08

Watchpoints, altering state, core files and threads

5 min read

Sometimes you know what is wrong but not who changes it. A answers "who wrote to this variable". Set one with watch, and add rwatch for a read-only watch and awatch for either access. Conditions attach the same way as breakpoints.

text
(gdb) watch reading            # stop when reading changes
(gdb) watch reading if reading > 100
(gdb) rwatch samples_taken     # stop when it is read

Watchpoints come in two very different kinds, and gdb reports which it installed. A hardware watchpoint uses the CPU's debug registers, four address registers on x86-64, so up to four are available at once, and it stops on the exact instruction that changes the value. gdb prints Hardware watchpoint 1: reading. If those registers are exhausted, or the target has none, gdb falls back to a software watchpoint, which single-steps the program and compares the value after every instruction, hundreds of times slower, and it reports the change at the next source line rather than the exact instruction. set can-use-hw-watchpoints 0 forces the software kind, which is how you tell which you have. In a multi-threaded program, hardware watchpoints watch the expression in every thread, while software watchpoints are effectively limited to the current thread, so on threaded code the hardware kind is essential.

Conditional breakpoints deserve one caution: gdb evaluates the condition by stopping at the breakpoint, testing the expression, and continuing invisibly if it is false. In a hot loop this makes the program crawl, and the hardware-assisted equivalent, if you only need a value to change, is often a watchpoint instead.

gdb can change the program while it runs. set var x = 5 assigns a new value, spelled set var to distinguish it from gdb's own set settings, and print x = 5 does the same because an assignment is also an expression. call func(args) invokes a function in the inferior and prints its result, letting you test a helper or force a code path without restarting. return expr makes the current function return immediately with a given value, and jump line resumes execution at a chosen line, both of which are for experiments, since they can corrupt state if misused.

A captures a crash for later. It is produced only if the shell's core limit allows it, raised with ulimit -c unlimited, and its name and location come from /proc/sys/kernel/core_pattern.

bash
ulimit -c unlimited
./monitor                 # segfaults, writes a core file
gdb ./monitor core        # inspect the crash

The first thing to do inside gdb is bt, which shows the exact call stack at the moment of the fault, then frame 0, info locals and print to read the values. On many systems the core is not a plain file: core_pattern may be a pipe into a handler such as systemd-coredump, and on Ubuntu 24.04 the cores are collected with coredumpctl rather than left in the working directory. cat /proc/sys/kernel/core_pattern shows which arrangement your system uses, and a value beginning with | means a handler program is receiving the dump. On an embedded board with no room for a core file, you instead rely on gdbserver and remote debugging, the subject of the next section, to catch the crash live.

Threads turn a single backtrace into many. info threads lists every thread with an id, and thread N switches the context of all the inspection commands to thread N. thread apply all bt prints a backtrace for every thread at once, the single most useful command when a multithreaded program is stuck or crashed, because it shows what every thread was doing. thread apply 2-3 bt restricts it to a range. A related setting, set scheduler-locking step, keeps the other threads stopped while you single-step, so a step does not get lost when the scheduler runs another thread in between.

Analysing a core dumpProgramKernelCore handlergdbfault(SIGSEGV)dump memory(core_pattern)gdb binarycorebt, frame,print1. Program → Kernelfault (SIGSEGV)2. Kernel → Core handlerdump memory (core_pattern)3. gdb → Core handlergdb binary core4. Core handler → gdbbt, frame, print
A crash writes a core file, and gdb plus the binary reconstructs the call stack and variable values
Hardware watchpoints are finite

On x86-64 only four hardware watchpoints exist, and each can watch at most eight bytes. A fifth watch silently becomes a software watchpoint, slowing the program by orders of magnitude and changing the stop precision. When gdb prints a watchpoint without the word hardware, you are in the slow path, and the fix is to free one of the four or watch a smaller region.

09

Threads, forks and remote debugging

4 min read

When a program calls fork, the kernel creates a child that is a copy of the parent. By default gdb keeps following the parent and lets the child run free, which is why a breakpoint placed in code the child executes can surprise you. set follow-fork-mode child switches gdb to follow the child instead, and set detach-on-fork off keeps both processes under gdb as separate inferiors, listed with info inferiors and selected with inferior N. On glibc, fork is implemented with the clone system call, so a program that creates processes is exercising the same machinery a threaded program does, and gdb presents both through the same inferior model. The catch command hooks the same events directly: catch fork, catch exec and catch syscall read stop when they occur.

Threads add the non-stop mode. Normally, when one thread hits a breakpoint, every thread stops. set non-stop on, which must be set before run, changes this so a continue advances only the current thread while the others keep running, and interrupt -a stops them all. This matters when a bug is a race that only appears while other threads are actually running, rather than frozen at a breakpoint.

Breakpoint command lists automate the repetitive parts. commands attaches a script to a breakpoint, run whenever it stops, and silent suppresses the normal announcement, so you can instrument a hot path without stopping.

text
(gdb) break update_sensor
Breakpoint 1 at 0x...: file sensor.c, line 42.
(gdb) commands 1
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>printf "reading = %d\n", reading
>continue
>end

Now every call to update_sensor prints its reading and resumes, turning gdb into a targeted, temporary logging tool that requires no edit to the source, which is exactly what you want on code you cannot easily rebuild on a target.

Remote debugging is the embedded workhorse. On the target, run the program under gdbserver, either over the network or a serial line:

bash
# on the target
gdbserver :2345 ./monitor          # network, wait for a host gdb on port 2345
gdbserver /dev/ttyS0 ./monitor     # serial link
gdbserver --attach :2345 1234      # attach to a running process
gdbserver --multi :2345            # no program yet; the host chooses one

On the host, load the binary you want to debug, tell gdb where the cross-compiled libraries live with set sysroot, and connect.

text
(gdb) file ./monitor
(gdb) set sysroot /opt/pi-rootfs
(gdb) target remote board:2345            # for a plain gdbserver
(gdb) target extended-remote board:2345   # for gdbserver --multi

target remote connects to a gdbserver that is already running one program. target extended-remote is the more capable form used with --multi, letting the host gdb start, kill and restart programs on the target over the same connection, and it is required if you did not name the program on the gdbserver command line. In that case, name it with set remote exec-file /path/on/target and then run. Serial debugging uses the same commands with the device instead of a host and port: target remote /dev/ttyUSB0.

Remote debugging with gdbserverHost gdbgdbserverProgramtarget remoteboard:2345spawn or attachbreak main, continueread/write memory,registerstrap, stopstop, registers1. Host gdb → gdbservertarget remote board:23452. gdbserver → Programspawn or attach3. Host gdb → gdbserverbreak main, continue4. gdbserver → Programread/write memory, registers5. Program → gdbservertrap, stop6. gdbserver → Host gdbstop, registers
The host gdb and the target gdbserver exchanging commands over a network socket or serial line
The network link carries symbols, not just commands

The target has the stripped binary and gdbserver; the host has the full binary with debugging information. gdbserver reads and writes target memory and registers, and gdb maps them back to source using its local copy. Keep the two in sync: the binary on the target and the one gdb loaded must be the same build, or addresses will not line up and breakpoints will land in the wrong place. This is what build-ids, introduced next, exist to verify.

10

DWARF, debuginfod, Python and reverse debugging

5 min read

The debugging information that -g produces is stored in a standardized format called . The -g flag is a level, not a switch: -g gives level 2, the default, with line numbers, variable locations and types. -g1 is the minimal level, enough for a backtrace but without locals or line-by-line detail, useful when you want a slightly smaller file. -g3 adds macro definitions, and -ggdb3 requests the gdb-specific extensions and the most information. -g0 disables it entirely.

Because debugging information is large, sometimes larger than the code it describes, shipping it to an embedded target is wasteful. The toolchain can split it out: objcopy --only-keep-debug monitor monitor.debug copies just the debugging sections into a separate file, strip --strip-debug monitor removes them from the shipped binary, and objcopy --add-gnu-debuglink=monitor.debug monitor records the name of the debug file inside the binary so gdb finds it automatically.

bash
gcc -g -o monitor main.c sensor.c
objcopy --only-keep-debug monitor monitor.debug
strip --strip-debug monitor
objcopy --add-gnu-debuglink=monitor.debug monitor
gdb ./monitor        # gdb loads monitor.debug via the debuglink

The modern mechanism that generalizes this is the build-id. The linker can embed a unique identifier for the exact build, gcc -Wl,--build-id ..., and readelf -n monitor prints it. Debug files are then stored by that id, under /usr/lib/debug/.build-id/xx/yyyy...debug, and gdb finds them no matter where the binary was moved, which is what makes it possible to debug a binary whose source and debug file you no longer have locally. This is how distribution packages ship debug information separately from the programs.

debuginfod closes the last gap: a server that serves debug information and source over HTTP, keyed by build-id. Set DEBUGINFOD_URLS=https://debuginfod.ubuntu.com and gdb downloads the missing debug file or source on demand when you open a binary, with no manual step. The command debuginfod-find debuginfo <build-id> fetches one directly. For embedded work this means you can keep the target lean and still debug a binary whose debug information lives on a server, not on the board.

gdb is scriptable in Python. A script can define new commands, subclass breakpoints to run code when they stop, and install pretty-printers that render library types readably.

python
import gdb

class PrintOpen(gdb.Breakpoint):
    "Print the pathname on every open() without stopping."
    def stop(self):
        path = gdb.parse_and_eval("(char *)$rdi")
        print("open(%s)" % path)
        return False   # do not stop the inferior

PrintOpen("open")

class SayHi(gdb.Command):
    "A custom command."
    def __init__(self):
        super().__init__("sayhi", gdb.COMMAND_USER)
    def invoke(self, arg, from_tty):
        print("hi " + arg)

SayHi()

Load it with source tool.py, then sayhi learner prints hi learner, and every open call logs its path without pausing the program. Pretty-printers, registered through the gdb.printing module, make print show a std::string or a kernel structure's fields instead of raw bytes, and convenience functions, subclasses of gdb.Function, add new functions you can call from print. This is the same interface the kernel's gdb helper scripts use to add Linux-aware commands.

Reverse debugging runs a program backwards. record (or record full) starts recording every instruction's effects, and once a crash or breakpoint is reached, you can step back through the events that led to it.

text
(gdb) break main
(gdb) run
(gdb) record
(gdb) continue               # run until the crash
(gdb) reverse-continue       # run backwards to the previous stop
(gdb) reverse-step           # step one line backwards
(gdb) reverse-next           # step one line backwards, over calls

record full is slow and memory-hungry because it logs the effect of every instruction, so it is for short, reproducible failures, and record btrace uses the CPU's branch-trace hardware where available for a lighter record. The separate tool rr, which records and replays whole programs out of process, is covered later in the module alongside sanitizers and perf.

Debug information pipelinegcc -gembed DWARFlink withbuild-idnotesectionobjcopysplit--only-keep-debugdebuginfodserve bybuild-idgcc -gembed DWARFlink withbuild-idnotesectionobjcopysplit--only-keep-debugdebuginfodserve bybuild-id
Debug information split from the binary, matched by build-id, fetched by debuginfod, and scripted from Python
Going deeperBuild-ids and the .build-id directory layout

A build-id is a short hash, usually an SHA-1, of the binary's contents, written by the linker into a .note.gnu.build-id section. The on-disk convention stores the debug file at .build-id/ab/cdef...debug, where the first two hex digits of the id form the subdirectory and the rest form the filename. gdb, debuginfod and tools such as eu-unstrip all locate files by this id, which is why a stripped binary copied to a different machine can still be matched to its debug information. --build-id is the default in current GCC on Ubuntu, so most binaries you build already carry one.

11

Common mistakes and how to fix them

3 min read

The mistakes are few and repeat. Learning to map a symptom to a cause is most of the skill.

The missing separator is make's most famous error, and it means a recipe line was indented with spaces instead of a tab. The fix is a literal tab, and an editor configured for Makefiles will insert one.

The stale build has no error at all, which is why it is the most dangerous. The symptom is that a source change has no effect, and the diagnosis is to suspect the dependency list: a header that a rule does not mention. The fix is -MMD -MP with -include so the compiler records the true dependencies, and the habit of never trusting a hand-written prerequisite list.

The -j race is the stale build's parallel sibling. It appears only sometimes, as a link that fails on a missing object or a binary that behaves inconsistently, because two recipes that should have been ordered ran concurrently. The cause is an incomplete dependency graph, and the fix is the same: make the graph correct first, then enable parallelism.

The optimized-out variable is a gdb symptom: print answers <optimized out> because the compiler kept the value in a register or eliminated it. The diagnosis is that the file was compiled with optimization, and the fix is to recompile just that file with -O0 -g while you debug it, leaving the rest optimized.

The wrong binary is a remote-debugging classic. Breakpoints land on the wrong lines or not at all, because the binary gdb loaded on the host is a different build from the one running under gdbserver on the target. The diagnosis is to compare the build-ids with readelf -n, and the fix is to copy the exact binary, not a similar one, and to keep the host's debug file in sync.

The missing core file has a simple cause: the core limit was zero, or a handler intercepted the dump. ulimit -c shows the limit, ulimit -c unlimited raises it for the shell, and cat /proc/sys/kernel/core_pattern shows whether a pipe handler such as systemd-coredump is receiving the core instead of a file being written. On Ubuntu 24.04 the dump may be in the journal under coredumpctl, not the working directory.

Common mistakes and their fixesmissingseparatorspaces nottabuse a tabstale buildheader nota prereq-MMD -MP-j raceincompletegraphfix depsfirstoptimizedout-O2 build-O0 -gwrong binarymismatchedbuildcomparebuild-idno core filelimit 0 orhandlerulimit -c,coredumpctlsymptom, cause, then fix
Six recurring build and debugging mistakes, their symptoms and their fixes
Do not reach for make clean first

A build that only works after make clean has a dependency bug, and cleaning before every build masks it while making every rebuild slow. When a clean build works but an incremental one misbehaves, treat it as a red flag and audit the dependency tracking rather than leaning on clean as a habit.

12

Try it

1 min read

Build a three-file sensor program, write a Makefile with variables, a pattern rule and a phony clean target, then migrate it to -MMD -MP with -include and prove that editing a header triggers a rebuild while an untouched object is skipped. Run the program under gdb, set a breakpoint, step and next through a function, print a structure, examine memory with x/, and read a backtrace with bt full and frame. Force a crash, raise the core limit, open the core with gdb, and reconstruct the failing line from the backtrace. Add a second thread, use info threads and thread apply all bt, and finally run the program under gdbserver on your board and connect from the host with target extended-remote. The practice pack extends each of these with a hardware-watchpoint exercise and a build-id and debuginfod walkthrough.

Key takeaways

  • make is a timestamp-driven dependency graph: a recipe runs only when a target is missing or older than a prerequisite.
  • Automatic variables $@, $<, $^ and $? plus pattern rules turn a page of repeated rules into a few generic lines.
  • =, :=, ?= and += differ in when expansion happens; CPPFLAGS, CFLAGS and LDFLAGS each carry a different kind of flag.
  • A Makefile that ignores headers goes silently stale; -MMD -MP with -include records the true dependency set.
  • gdb maps machine state back to source through DWARF; break, print, x/, bt and frame are the core loop.
  • Hardware watchpoints run in CPU debug registers and stop on the exact instruction; software watchpoints single-step and are far slower.
  • A core file plus the matching binary reconstructs the crash without a live debugger.
  • gdbserver splits gdb across host and target, the standard way to debug a headless embedded board.

Reflect

  1. Explain to a colleague, in two sentences: explain why separate compilation needs a build system and what guarantee make's timestamp-based graph provides.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close