Module 6 · Lesson 2 of 24

Building applications for the target (CMake, SDK, remote debug)

Cross-build an application with CMake and an SDK, deploy it to the target, and debug it remotely with gdbserver and gdb, then script the whole loop.

  • Beginner to advanced
  • 26 min read
  • 50 min lab
What you will need

Ubuntu 24.04 (WSL2, VM or container) with a cross toolchain or a Yocto SDK; a QEMU aarch64 target or a Raspberry Pi for the debug sections

By the end you can 8 objectives
  • Explain the difference between host and target compilation and choose the right one for a given task
  • Write a CMake toolchain file and a CMakePresets.json that cross-build for an aarch64 target
  • Resolve a target library through pkg-config and find_package without leaking host paths
  • Structure a project with modern CMake targets and PUBLIC, PRIVATE and INTERFACE scope
  • Deploy a binary with rsync, NFS or sshfs and wrap deployment in a CMake custom target
  • Attach host gdb to gdbserver, set a sysroot, and use breakpoints and watchpoints on a cross-built binary
  • Split debug information from a stripped binary and evaluate where each copy should live
  • Design a one-command build, deploy and debug loop and diagnose its common failures
Read the lesson below; the video will appear here when published.
01

Host builds, target builds and the sysroot

3 min read

The previous lesson split a product into layers and services. This lesson is about getting your own code, the application layer, onto the device and keeping it debuggable. Every step starts with one question: which machine runs the compiler, and which machine runs the result?

The host is the machine you work on, usually an x86-64 Ubuntu 24.04 system. The target is the device the product runs on, often 32-bit ARM or AArch64. When both are the same architecture, you do a . When they differ, you do a .

Each toolchain is named by a . Common ones are aarch64-linux-gnu (64-bit ARM, glibc), arm-linux-gnueabihf (32-bit ARM with hardware floating point), and riscv64-linux-gnu. Yocto uses a vendor field, as in aarch64-poky-linux. The compiler binaries carry the triplet as a prefix, and gcc -dumpmachine prints the triplet a compiler targets.

Compiling alone is not enough. The program must link against the target's C library and any other libraries, using headers that describe the target's ABI. A supplies them. The compiler flag --sysroot=DIR tells GCC to treat DIR as the root when it looks up system headers and libraries, so #include <stdio.h> is read from DIR/usr/include and -lc is found in DIR/usr/lib.

Two conventions you will meet constantly. First, CROSS_COMPILE is a prefix variable used by the Linux kernel, U-Boot, Buildroot packages and many Makefiles: make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 makes the build call aarch64-linux-gnu-gcc. It is a convention of those build systems, not something GCC itself reads. Second, autotools projects take --host=aarch64-linux-gnu to say what the output will run on, while --build names the machine doing the compiling.

When is each style correct? Build natively when the target is fast enough and has a compiler, such as a Raspberry Pi compiling a small tool; the build is trivially correct because host and target match. Cross-build when the target is slow, has little storage or RAM, has no compiler in its image, or when you build many times a day: a desktop is far faster than a small ARM core. Product images are always cross-built, because the production image never ships a compiler. A third option is running target binaries on the host with qemu-user, which translates one instruction set to another and is handy for quick tests, though it does not model real hardware.

Native build versus cross buildNative buildCross buildCompiler runs onthe targetitselfthe host PCOutput runs onthe samemachinea differenttargetLibraries fromthe system/usra targetsysrootBest forsmall toolson fastboardsimages anddaily builds
Native build versus cross build: where the compiler runs and what it produces
02

The two SDK families: Yocto and Buildroot

4 min read

You can install a generic cross-compiler from Ubuntu, for example the gcc-aarch64-linux-gnu package, and it works for simple programs. The trouble starts when your image uses a different glibc version, a different set of libraries, or a different CPU tuning than the packaged compiler assumed. The robust answer is to build the SDK from the same recipe set that builds the image, so compiler, headers and libraries all match. Two families dominate.

The Yocto SDK. Running bitbake <image> -c populate_sdk produces a self-extracting installer script, named after the distro, image and machine, ending in -toolchain-<version>.sh. It installs under /opt/poky/<version> by default, or any directory you choose. The installer contains a native-side half that runs on your host (compiler, linker, CMake, gdb) and a target-side half, the target sysroot. The standard SDK is fixed once built. The extensible SDK, built with populate_sdk_ext, also contains a trimmed copy of the build system and the devtool command, which we meet in the advanced section. Check the release you use for the current naming; the Yocto Project cuts a new release every six months and designates some as long-term support.

After installing, you do not just add a directory to your PATH. You source an environment script:

bash
. /opt/poky/<version>/environment-setup-cortexa57-poky-linux
echo "$CC"
$CC --version

The script name encodes the tuning of the target machine. Sourcing it exports a set of variables. The important ones are CC, CXX, CPP, LD, AR, STRIP, OBJCOPY, OBJDUMP and GDB, each pointing at the cross tool with the right prefix; CROSS_COMPILE; SDKTARGETSYSROOT and OECORE_TARGET_SYSROOT, naming the sysroot; OECORE_NATIVE_SYSROOT, naming the host-side tools; PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH; and flag variables such as CFLAGS, CXXFLAGS and LDFLAGS carrying the tuning and hardening options chosen by the distro. One detail catches people out: $CC is not a bare program name. It normally includes machine flags and the --sysroot= option, so it is a command line, and quoting it as a single path breaks things.

The Buildroot toolchain. Buildroot builds a toolchain into its output tree, and the layout is regular: the host tools live in output/host/bin with the triplet prefix, and the sysroot sits in output/host/<triplet>/sysroot. Running make sdk packs this into a relocatable tarball in output/images, and an included script, relocate-sdk.sh, fixes the embedded paths after you unpack it somewhere new. Buildroot also generates a CMake toolchain file, typically at output/host/share/buildroot/toolchainfile.cmake, and a pkg-config wrapper that already points at the target. Buildroot's toolchain can be built internally or taken from an external prebuilt one; the manual describes the options and their trade-offs.

The mental model is the same in both: put the SDK tools first on PATH, point the build at the sysroot, and let nothing from the host's /usr/include or /usr/lib leak in. Everything else in this lesson is about enforcing that last clause.

Step by stepFrom SDK installer to a target binary
  1. Build the SDK: the image build produces an installer script that matches the image's libraries

  2. Install: run the script on the host and choose an install directory

  3. Source the environment: the environment-setup script exports the tool paths, sysroot and flags

  4. Configure: CMake or make reads those variables and selects the cross tools

  5. Compile and link: the cross compiler builds against the target sysroot, never the host libraries

  6. Check the result: run file on the binary and confirm the architecture is the target's

Yocto SDK and Buildroot toolchainYocto SDKBuildroottoolchainProduced bybitbakepopulate_sdkmake sdkSet up withinstaller,then sourcescriptunpack, run relocate-sdk.shTools live innativesysroot under/opt/pokyoutput/host/binTarget librariesintargetsysroottripletsysrootdirectoryCMake fileOEToolchainConfig.cmaketoolchainfile.cmake
Where the Yocto SDK and the Buildroot toolchain each place the cross tools and the sysroot
03

CMake for cross builds: toolchain files and presets

4 min read

does not compile anything itself. For cross builds, it needs to be told that the target differs from the host. That information goes in a . Here is a hand-written one for an AArch64 board, saved as cmake/aarch64-linux-gnu.cmake:

cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(TRIPLET aarch64-linux-gnu)
set(CMAKE_C_COMPILER   ${TRIPLET}-gcc)
set(CMAKE_CXX_COMPILER ${TRIPLET}-g++)

set(CMAKE_SYSROOT /home/learner/sysroots/board)

set(CMAKE_FIND_ROOT_PATH ${CMAKE_SYSROOT})
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)

Each line has a reason. Setting CMAKE_SYSTEM_NAME yourself tells CMake it is cross-compiling, and it then treats the target as different from the host. CMAKE_SYSTEM_PROCESSOR is not detected for you in a cross build; you state it, and your own CMake code can branch on it. The two compiler lines choose the tools, with the triplet prefix found on PATH. CMAKE_SYSROOT makes CMake pass --sysroot to both compile and link steps.

The CMAKE_FIND_ROOT_PATH block is the heart of correctness. find_library, find_path, find_file and find_package normally search the host's /usr. The root path list says "also look under the sysroot", and the mode variables say how strict to be. ONLY means search the sysroot and nothing else, which is what you want for libraries, headers and packages. NEVER for programs means that find_program keeps searching the host, which is right because build-time tools such as Python, protoc or a code generator must run on the host. The default, BOTH, quietly allows a host library to satisfy a target dependency, and that produces binaries that fail on the board with a wrong-architecture link error or, worse, link against the host's version of a header.

Setting CMAKE_TRY_COMPILE_TARGET_TYPE to STATIC_LIBRARY makes CMake's internal compiler checks stop after compiling, so they do not need a working link against the sysroot's start files during configuration. That avoids a classic failure where configure dies at "The C compiler is not able to compile a simple test program". If the target runs under qemu-user, CMAKE_CROSSCOMPILING_EMULATOR can hold the emulator command so tests and generated-code steps run, for instance qemu-aarch64;-L;/home/learner/sysroots/board.

With a Yocto SDK you do not write this file. The SDK ships one, and the documented way is to pass -DCMAKE_TOOLCHAIN_FILE=$OECORE_NATIVE_SYSROOT/usr/share/cmake/OEToolchainConfig.cmake. With Buildroot, pass the generated toolchainfile.cmake. Either way, the toolchain file belongs to the SDK, not to your source tree, so your CMakeLists.txt should never name an absolute compiler path.

Typing long -D options every time is where mistakes creep in. CMake presets, kept in a JSON file named CMakePresets.json, name reusable configure, build and test settings so a long command line becomes one preset name, and they store them in the repository:

json
{
  "version": 3,
  "cmakeMinimumRequired": { "major": 3, "minor": 21, "patch": 0 },
  "configurePresets": [
    { "name": "base", "hidden": true, "generator": "Ninja",
      "binaryDir": "${sourceDir}/build/${presetName}",
      "cacheVariables": { "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" } },
    { "name": "host-debug", "inherits": "base",
      "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } },
    { "name": "arm64-debug", "inherits": "base",
      "toolchainFile": "${sourceDir}/cmake/aarch64-linux-gnu.cmake",
      "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } },
    { "name": "yocto", "inherits": "base",
      "toolchainFile": "$env{OECORE_NATIVE_SYSROOT}/usr/share/cmake/OEToolchainConfig.cmake",
      "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } }
  ],
  "buildPresets": [
    { "name": "arm64-debug", "configurePreset": "arm64-debug" }
  ]
}

Then cmake --preset arm64-debug configures and cmake --build --preset arm64-debug builds, with identical results on every engineer's machine and in CI. The toolchainFile field needs preset schema version 3 or later, and the file is committed, so the team shares one definition. Personal tweaks go in CMakeUserPresets.json, which you keep out of version control.

From preset to target binaryPick presetcmake--presetarm64-debugLoadtoolchainfilecompilers,sysroot,find modesConfigurefindlibrariesin sysrootBuildNinja runscrosscompilerCheckbinaryfile showstargetmachinePick presetcmake--presetarm64-debugLoadtoolchainfilecompilers,sysroot,find modesConfigurefindlibrariesin sysrootBuildNinja runscrosscompilerCheckbinaryfile showstargetmachine
How a toolchain file and a preset steer CMake from configure to a target binary
04

Modern CMake: targets, scope, build types and installation

4 min read

Old CMake used global variables and commands such as include_directories that affected everything. Modern style, which means CMake 3.x practice, attaches settings to a target, meaning a named build product such as an executable or library that carries its own properties, and lets CMake propagate them. Here is a small SensorNode project with a library and an application, using SQLite as the single external dependency:

cmake
cmake_minimum_required(VERSION 3.21)
project(sensornode VERSION 0.1.0 LANGUAGES C)

include(GNUInstallDirs)
find_package(PkgConfig REQUIRED)
pkg_check_modules(SQLITE3 REQUIRED IMPORTED_TARGET sqlite3)

add_library(sensor STATIC src/sensor.c)
target_include_directories(sensor PUBLIC include)
target_compile_features(sensor PUBLIC c_std_11)
target_link_libraries(sensor PRIVATE PkgConfig::SQLITE3)

add_executable(sensornode src/main.c)
target_link_libraries(sensornode PRIVATE sensor)

install(TARGETS sensornode RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})

The three scope keywords are the part to memorise:

  • PRIVATE: the requirement applies only when building this target. Nobody who links to it inherits it.
  • INTERFACE: the requirement applies only to targets that use this one, not to itself. A header-only library has nothing but interface requirements.
  • PUBLIC: both. The target needs it, and so does everyone who links to it.

In the example, include is PUBLIC for sensor because its header appears in the signature of functions that sensornode calls, so sensornode needs that directory too, and CMake provides it automatically through target_link_libraries(sensornode PRIVATE sensor). SQLite is PRIVATE because only sensor.c uses it; if a public header exposed a sqlite3 * type, the dependency would have to be PUBLIC. Getting scope right keeps link lines short and stops a change in one library from rebuilding the world.

target_compile_features(sensor PUBLIC c_std_11) states a language requirement instead of hard-coding -std=gnu11; CMake adds whatever flag the selected compiler needs. include(GNUInstallDirs) defines the standard destination names, CMAKE_INSTALL_BINDIR, CMAKE_INSTALL_LIBDIR and so on, so install(TARGETS ...) places files at bin and lib relative to the install prefix, and distribution multiarch variations are handled for you. To stage into a directory instead of the live root, use DESTDIR:

bash
DESTDIR=/home/learner/stage cmake --install build/arm64-debug --prefix /usr

That is how a build system or an NFS root gets a tree it can copy straight onto the target. Running cmake --install build/arm64-debug --prefix /usr with a DESTDIR environment variable writes /home/learner/stage/usr/bin/sensornode.

Build types. CMAKE_BUILD_TYPE, the cache variable that selects default compiler flags for single-configuration generators such as Makefiles and Ninja, defaults to empty, which means no optimisation and no debug flags. The standard values add: Debug, -g; Release, -O3 -DNDEBUG; RelWithDebInfo, -O2 -g -DNDEBUG; and MinSizeRel, -Os -DNDEBUG. For a target the usual choice is RelWithDebInfo while developing, because it matches what ships yet keeps symbols, and Release or MinSizeRel for the final image with the symbols split off. Target-specific tuning, such as -mcpu=cortex-a72, belongs in the toolchain file's CMAKE_C_FLAGS_INIT, since the SDK already carries the flags for its machine.

Generators and caching. Choose Ninja with -G Ninja or a preset generator; it has less overhead than Make and keeps all cores busy. Add ccache by setting CMAKE_C_COMPILER_LAUNCHER=ccache (and the C++ equivalent): the launcher wraps each compiler call, hashes the source, headers and flags, and returns a stored object file on a hit. Because cross builds repeat identical work after clean or a branch switch, the speed-up is substantial.

PRIVATE, INTERFACE and PUBLICPRIVATEINTERFACEPUBLICTarget itselfuses ityesnoyesConsumersinherit itnoyesyesTypical useinternaldependencyheader-onlylibrarytype inpublic header
PRIVATE, INTERFACE and PUBLIC: who inherits a usage requirement
05

Finding target libraries: pkg-config, find_package and third-party code

5 min read

is how most C libraries advertise themselves. A sqlite3.pc file records the prefix, the version, Cflags: -I${includedir} and Libs: -L${libdir} -lsqlite3. Run natively, it reads the host's .pc files. Run for a cross build, three environment variables control it, and the difference between them matters:

  • PKG_CONFIG_PATH adds directories to the front of the default search list. It does not remove the host directories.
  • PKG_CONFIG_LIBDIR replaces the default search list. This is the one that keeps host libraries out.
  • PKG_CONFIG_SYSROOT_DIR is prepended to the -I and -L paths in the output. A .pc file inside the sysroot says /usr/include; with the sysroot variable set, the printed flag becomes -I/home/learner/sysroots/board/usr/include.

A correct cross environment therefore sets PKG_CONFIG_LIBDIR to the sysroot's usr/lib/pkgconfig, usr/lib/<multiarch>/pkgconfig and usr/share/pkgconfig, and PKG_CONFIG_SYSROOT_DIR to the sysroot. The Yocto environment script sets the sysroot and path variables; Buildroot provides a wrapper that does the equivalent. Some toolchains instead ship a prefixed wrapper script with these variables baked in.

CMake reaches pkg-config through the FindPkgConfig module. pkg_check_modules(SQLITE3 REQUIRED IMPORTED_TARGET sqlite3) runs the pkg-config found at configure time with CMake's environment, and IMPORTED_TARGET creates PkgConfig::SQLITE3, a target that carries the include and link flags. FindPkgConfig does not read CMAKE_SYSROOT for you, so the pkg-config variables must be exported by the SDK script or set from the toolchain file, for example set(ENV{PKG_CONFIG_SYSROOT_DIR} ${CMAKE_SYSROOT}) and set(ENV{PKG_CONFIG_LIBDIR} ...).

find_package(Foo) works differently. In module mode CMake runs a FindFoo.cmake script, which calls find_library and find_path; in config mode it loads a FooConfig.cmake or foo-config.cmake file installed by the library. In both modes the find-root settings from the toolchain file decide where the search may look, which is why the ONLY modes matter. A config file installed into the sysroot works well, because it records relative locations. A config file that hard-codes absolute host paths is broken for cross builds; the symptom is a build that finds the package and then fails with missing files.

In this sysroot, libz.so is a symlink with an absolute target, /usr/lib/libz.so.1. Resolved from the host root it points outside the sysroot and does not exist there, so the lookup fails. Real SDK sysroots rewrite such links to be relative for this reason.

  • cwd: /opt/sysroot/usr/lib
  • value: libz.so

The widget shows a real cross-build trap. A library symlink inside the sysroot that holds an absolute target resolves against the host's root, not the sysroot, so it dangles. Well-made SDKs rewrite those links to be relative; hand-copied sysroots often do not, and the symptom is "cannot find -lz" even though the file is right there.

When the SDK lacks a library. A cross build needs a library the SDK does not ship. You have four options, in descending order of how well they fit a product:

  1. Add it to the image and rebuild the SDK. In Yocto, write or reuse a recipe, add it to the image, and add the -dev package to the SDK with TOOLCHAIN_TARGET_TASK:append = " mylib-dev"; in Buildroot, enable or write a package. The SDK and the image then agree, which is the property you want.
  2. Use the extensible SDK. devtool add brings a new recipe into the workspace, covered later.
  3. Build it yourself into a private prefix. Cross-build the library with the SDK, install it to a directory such as /home/learner/deps/aarch64, and add that directory to CMAKE_FIND_ROOT_PATH and CMAKE_PREFIX_PATH plus the pkg-config path. You must remember to ship the shared library to the target too.
  4. Vendor the source. Copy the dependency's source into third_party/ and build it with add_subdirectory. This is the most reproducible choice for small libraries, and it carries a licence and security-patch burden you now own.

FetchContent downloads and builds a dependency at configure time. It is convenient on a desk and awkward in product builds. Yocto and Buildroot want sources fetched in a controlled step with checksums, and recent Yocto releases isolate network access during compile tasks, so a configure step that downloads fails there. If you use it, pin an exact tag or commit, and set FETCHCONTENT_FULLY_DISCONNECTED=ON or FETCHCONTENT_SOURCE_DIR_<NAME> for offline and packaged builds. FetchContent_Declare(... FIND_PACKAGE_ARGS) (CMake 3.24) tries find_package first and falls back to downloading, which lets a distribution build use the system copy. Prefer find_package for anything the image already ships.

Library name to sysroot flagsLibrarynamesqlite3Search .pcfilesonly PKG_CONFIG_LIBDIRdirsRead Cflagsand LibsPrefixsysrootPKG_CONFIG_SYSROOT_DIRExit from step 2: host dirs searchedwrong-arch libraryFlags tocompilerLibrarynamesqlite3Search .pcfilesonly PKG_CONFIG_LIBDIRdirsExit from step 2: host dirssearchedwrong-arch libraryRead Cflagsand LibsPrefixsysrootPKG_CONFIG_SYSROOT_DIRFlags tocompiler
How a library name becomes sysroot-correct compiler flags
06

Deploying to the target

2 min read

Four methods cover nearly every setup, each with a clear price.

scp and rsync over SSH. scp copies whole files. rsync -avz --partial -e ssh build/arm64-debug/sensornode learner@board:/opt/sensornode/ copies only blocks that changed, so after a small rebuild a few kilobytes cross the wire. One detail bites on embedded targets: since OpenSSH 9.0 the scp command speaks the SFTP protocol by default, and a target running Dropbear without an SFTP server rejects it. Ubuntu 24.04 ships OpenSSH 9.6, so scp -O (legacy protocol) is the workaround, or use rsync over a shell session. rsync must also exist on the target, which minimal BusyBox images often lack; then fall back to scp -O or to ssh board 'cat > file' < file.

NFS root filesystem. The host exports a directory, and the board mounts it as its root, so the target runs whatever is in the directory on the host. The kernel needs NFS root support, and the kernel command line carries the details:

bash
# /etc/exports on the host
/srv/nfs/board  192.168.7.0/24(rw,sync,no_subtree_check,no_root_squash)

# kernel command line on the board
root=/dev/nfs nfsroot=192.168.7.1:/srv/nfs/board,vers=3 ip=dhcp rw

no_root_squash lets root on the board own files as root, which a root filesystem needs, and which is why this belongs on a private development network only. Then DESTDIR=/srv/nfs/board cmake --install build/arm64-debug --prefix /usr updates the board's filesystem directly: no copy step at all, and nothing to flash. NFS root is the fastest loop for application and library work, and the wrong choice for anything needing a writable flash behaviour test.

sshfs. The host mounts a directory from the target (or the target mounts one from the host) through SSH and FUSE. It needs the SFTP subsystem on the server and a FUSE package on the client, so it suits Raspberry Pi class boards more than a stripped image. It is useful for browsing logs and editing config files on the board from the host, and slower than NFS for build output.

The image itself. Production deployment is not a copy at all but an OTA update, the topic of an earlier module. The three methods above are for the development loop only; never let a dev-only mechanism reach the product image.

You can fold deployment into the build so that one target does it every time. A CMake custom target runs commands on demand:

cmake
set(DEPLOY_HOST "board" CACHE STRING "Target host name or address")
set(DEPLOY_USER "learner" CACHE STRING "Login on the target")
set(DEPLOY_DIR  "/opt/sensornode" CACHE PATH "Install directory on the target")

add_custom_target(deploy
  COMMAND rsync -az --partial $<TARGET_FILE:sensornode>
          ${DEPLOY_USER}@${DEPLOY_HOST}:${DEPLOY_DIR}/
  DEPENDS sensornode
  COMMENT "Copying sensornode to ${DEPLOY_HOST}"
  VERBATIM)

$<TARGET_FILE:sensornode> is a generator expression that expands to the real path of the built file for whichever generator and build type is in use, and DEPENDS sensornode makes cmake --build build --target deploy rebuild first. VERBATIM makes CMake quote the arguments correctly for the shell. Remember that a custom target always runs when requested; it is not tied to the default all target unless you add ALL, and you should not, because a normal build must not need a reachable board.

Going deeperWhy scp -O exists, and key handling for the dev loop

The old scp protocol ran a remote scp -t process and exchanged a stream shaped by the remote shell. That design had a history of quoting and path-handling vulnerabilities, because the server executed what the client sent. OpenSSH 9.0 therefore moved scp onto SFTP internally, keeping the command name. SFTP needs an sftp-server on the target. Dropbear, common in small images, does not include one by default, which is why the flag -O appears in so many embedded how-to guides. For an unattended loop, use a dedicated key pair for the board, add it with ssh-copy-id, and keep the board's host key in known_hosts; avoid disabling host-key checking globally, because a board that gets reflashed with a new key should raise a warning rather than hide a man-in-the-middle.

Ways to deploy to the boardrsync overSSHNFS rootsshfsCopieschangedblocksnothingfiles ondemandNeeds on targetssh server,rsyncNFS rootkernelsftp serverLoop speedfastfastestmoderateRiskold scpprotocolno_root_squash exposureneeds FUSE onclient
Four ways to get a build onto the board and what each costs
07

Remote debugging with gdbserver

5 min read

Debugging the target process directly would need a full gdb, your source tree and symbol files on a small board. splits the work. The target holds the stripped binary and gdbserver. The host holds the unstripped binary with symbols, the source files and the SDK's gdb.

On the host, use a gdb that understands the target architecture. The Yocto SDK exports $GDB pointing at its cross gdb, and Ubuntu offers gdb-multiarch, which handles many architectures in one binary (Ubuntu 24.04 packages gdb 15). A plain x86 gdb will not understand AArch64 registers.

Step by stepA remote debug session
  1. Start gdbserver: on the target, run it with a port and the program, so it launches the process and halts at its first instruction

  2. Start host gdb: run the cross gdb on the unstripped binary so it loads symbols

  3. Set the sysroot: tell gdb where the target's libraries are on the host, so it reads shared-library symbols locally

  4. Connect: target remote opens the TCP connection to gdbserver and gdb takes control

  5. Set breakpoints: gdb sends breakpoint requests; gdbserver plants them in the target process

  6. Continue and inspect: the target runs until a stop; gdb reads registers and memory and prints variables

  7. Detach or kill: end the session, leaving the target running or stopping it

On the target, start the program under gdbserver:

bash
gdbserver :2345 /opt/sensornode/sensornode

Typical output (yours will differ in process id and address) looks like Process /opt/sensornode/sensornode created; pid = 612 and Listening on port 2345, and later Remote debugging from host 192.168.7.1. The leading colon means listen on all interfaces. To debug an already running process instead, use gdbserver --attach :2345 612. For a session that survives the program exiting, gdbserver --multi :2345 lets the host choose and launch programs with target extended-remote.

On the host, a session looks like this:

bash
gdb-multiarch build/arm64-debug/sensornode
text
(gdb) set sysroot /home/learner/sysroots/board
(gdb) target remote 192.168.7.2:2345
(gdb) break sensor_read
(gdb) continue
(gdb) bt
(gdb) print r
(gdb) watch r.value
(gdb) continue

set sysroot matters more than it looks. When gdb connects to a remote target and no sysroot is set, it defaults to the special value target:, meaning "fetch shared libraries from the target over the remote connection". That works but is slow, and it reads the stripped libraries on the board, so you lose symbols for libc. Pointing the sysroot at the SDK or build tree's copy gives gdb the full-symbol libraries on the host. set solib-search-path adds extra directories for libraries that live outside the sysroot layout.

Breakpoints and watchpoints. A software breakpoint replaces an instruction with a trap; gdb restores it when you continue. hbreak asks for a hardware breakpoint, useful in read-only code. A uses the CPU's debug registers when it can: watch r.value stops on a write, rwatch on a read, awatch on either. Each ARM core offers only a handful of such registers, so large or numerous watches fall back to single-stepping, which is slow over a network link. show can-use-hw-watchpoints reveals the setting.

If you cannot open a port, tunnel gdbserver through SSH and avoid exposing it: target remote | ssh learner@board gdbserver - /opt/sensornode/sensornode. The pipe form runs the remote command and speaks the protocol over its standard input and output, so no listening port exists. gdbserver has no authentication of its own, so on a real network always use this form or an SSH port forward.

A gdbserver session moves from No session to gdbserver listening, then Connected and halted after target remote. Continue runs the target; a breakpoint hit halts it again, and program exit ends the session. Detach from the halted state leaves the program running free.

  • states: [{"id": "stopped", "label": "No session"}, {"id": "listening", "label": "gdbserver listening"}, {"id": "connected", "label": "Connected, halted"}, {"id": "running", "label": "Target running"}, {"id": "ended", "label": "Program exited"}]
  • transitions: [{"from": "stopped", "to": "listening", "event": "start gdbserver"}, {"from": "listening", "to": "connected", "event": "target remote"}, {"from": "connected", "to": "running", "event": "continue"}, {"from": "running", "to": "connected", "event": "breakpoint hit"}, {"from": "running", "to": "ended", "event": "program exits"}, {"from": "connected", "to": "stopped", "event": "detach"}]
A gdbserver sessionHost gdbgdbserverTarget processtarget remoteinsert breakpointptrace: plant trapcontinuestopped atbreakpointstop replyread registers,memory1. Host gdb → gdbservertarget remote2. Host gdb → gdbserverinsert breakpoint3. gdbserver → Target processptrace: plant trap4. Host gdb → gdbservercontinue5. Target process → gdbserverstopped at breakpoint6. gdbserver → Host gdbstop reply7. Host gdb → gdbserverread registers, memory
Messages between host gdb, gdbserver and the debugged process
08

Debug symbols, split debug info and IDE integration

4 min read

Compiling with -g adds DWARF sections to the object file that map addresses to source lines and describe types and variables. These sections are not loaded into memory when the program runs, so -g does not slow execution. It does enlarge the file on disk, often by several times. The real cost to debugging comes from the optimiser. At -O2 the compiler inlines functions, reorders statements and keeps variables in registers, so gdb reports <optimized out> and steps appear to jump around. -Og optimises while preserving the debugging experience and is a good development level; -O0 is clearest but may behave differently from the shipping build, which matters for timing bugs. RelWithDebInfo keeps -O2 -g so you debug what ships.

stripping removes the symbol table and debug sections from an ELF file, and produces the small binary that goes on the board. Do it with the cross tools, since the host strip does not understand foreign formats well. The split-debug workflow keeps both halves:

bash
aarch64-linux-gnu-objcopy --only-keep-debug sensornode sensornode.debug
aarch64-linux-gnu-strip --strip-all sensornode
aarch64-linux-gnu-objcopy --add-gnu-debuglink=sensornode.debug sensornode

The first line extracts the debug sections into sensornode.debug; the second strips the original; the third adds a .gnu_debuglink section naming the debug file and its CRC. gdb finds the pair through that name. A more robust link is the . GCC on Ubuntu passes --build-id to the linker by default, and readelf -n sensornode shows it. Debug files laid out as .build-id/ab/cdef....debug under a debug directory match the binary exactly, and gdb searches debug-file-directory for them. A debug file from a different build is the single biggest source of confusing sessions, with lines that do not match the source. Keep the debug file of every released image, named by its build-id, in an archive.

When symbols are not on your disk, fetches them. gdb reads the DEBUGINFOD_URLS environment variable. Ubuntu's public service covers Ubuntu's own packages, not your product. For your product, run debuginfod from elfutils over the directory that holds your released debug files, and every developer's gdb gets the right symbols on demand. The Yocto Project documents a public debuginfod service for its own release builds, which is useful when you debug a stock reference image.

From the editor. The command-line flow above is the foundation, and an IDE drives the same pieces. In Visual Studio Code, the CMake Tools extension reads CMakePresets.json so choosing arm64-debug configures and builds with the toolchain file. A launch.json entry using the C/C++ extension's cppdbg type attaches to a gdbserver already running on the board:

json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug on board",
      "type": "cppdbg",
      "request": "launch",
      "program": "${command:cmake.launchTargetPath}",
      "cwd": "${workspaceFolder}",
      "MIMode": "gdb",
      "miDebuggerPath": "/usr/bin/gdb-multiarch",
      "miDebuggerServerAddress": "board:2345",
      "setupCommands": [
        { "text": "set sysroot /home/learner/sysroots/board" }
      ]
    }
  ]
}

program is the unstripped host binary, miDebuggerServerAddress is the gdbserver address, and setupCommands runs the same set sysroot you would type by hand. A tasks.json entry that runs your deploy script and starts gdbserver can be hooked up through preLaunchTask, so F5 builds, copies and debugs. The IDE is a front end; if it fails, the same commands in a terminal tell you why.

Going deeperWhat is inside the ELF debug link, and why stripping is safe

Stripping removes the .symtab and .strtab sections and all .debug_* sections. None are in a loadable segment, so the program's memory image is identical before and after: the loader maps only segments from the program header table, and the section headers are for tools. The dynamic symbol table, .dynsym, stays, because the dynamic linker needs it to resolve shared-library symbols. That is why a stripped binary still runs and still shows exported function names in a backtrace, while local functions appear only as addresses. If you use strip --strip-unneeded on a shared library, it keeps .dynsym and removes the rest. The .gnu_debuglink section holds a file name and a CRC32 of the debug file's contents, so gdb can reject a stale file; the build-id note is stronger because it identifies the linked output rather than the debug file.

Splitting debug informationLinkedbinarycode plusDWARFExtractdebug fileobjcopy --only-keep-debugStripbinarysmall copyfor theboardAdddebuglinkname andCRCMatch bybuild-idgdb pairsthe twoLinkedbinarycode plusDWARFExtractdebug fileobjcopy --only-keep-debugStripbinarysmall copyfor theboardAdddebuglinkname andCRCMatch bybuild-idgdb pairsthe two
One build, two files: a stripped binary for the board and a debug file for the host
09

The one-command loop, reproducible builds and caches

3 min read

The edit, build, deploy, debug cycle repeats hundreds of times a day, so each manual step costs you. Collect them in one script, tools/dbg.sh:

bash
#!/usr/bin/env bash
set -euo pipefail

TARGET="${TARGET:-learner@board}"
PORT="${PORT:-2345}"
PRESET="${PRESET:-arm64-debug}"
REMOTE_DIR="${REMOTE_DIR:-/opt/sensornode}"
SYSROOT="${SYSROOT:-/home/learner/sysroots/board}"
BIN="sensornode"

cmake --preset "$PRESET"
cmake --build --preset "$PRESET"

BUILD="build/$PRESET"
rsync -az --partial "$BUILD/$BIN" "$TARGET:$REMOTE_DIR/"

exec gdb-multiarch -q "$BUILD/$BIN" \
  -ex "set sysroot $SYSROOT" \
  -ex "target remote | ssh $TARGET gdbserver - $REMOTE_DIR/$BIN" \
  -ex "break main" -ex "continue"

Each choice is deliberate. set -euo pipefail stops at the first failure, so a compile error never reaches the board. The pipe form of target remote starts gdbserver over SSH and needs no port or race between starting the server and connecting. Environment variables with defaults let a teammate or a CI job override the board address without editing the file. The same logic can be exposed as a CMake custom target named debug, but a script is easier to read, and a CMake target is better when the IDE should trigger it. Note the stripped-binary rule from before: the host copy in build/ has symbols, while the copy on the board may be stripped to match production, since gdb reads symbols from the host side.

Reproducible builds. A build is when nothing environmental leaks into the output. Three leaks are common. Timestamps: __DATE__, __TIME__ and archive members carry the build time. Set SOURCE_DATE_EPOCH to a fixed Unix time (many teams use the last commit time with git log -1 --format=%ct), and GCC substitutes it for the macros. Paths: GCC embeds the absolute build directory in debug info and in __FILE__, so two checkouts in different directories differ. Use -ffile-prefix-map=/home/learner/work/sensornode=. (GCC 8 and newer) to rewrite the prefix in both places. Ordering: parallel builds and filesystem iteration order can reorder inputs; modern linkers and archivers are deterministic for the same inputs. Tools such as diffoscope compare two outputs and show where they differ.

The benefit for the loop is practical: with path maps and a fixed epoch, a preprocessed translation unit does not change between checkouts, so the compiler cache hits. ccache keys on the preprocessed text and flags; setting hash_dir = false in its config stops the working directory from entering the hash when you use debug prefix maps, and base_dir rewrites absolute paths under it to relative ones. sccache is a similar wrapper that can store results on a shared server, such as an S3-compatible bucket, so a whole team shares hits. Both are enabled through CMAKE_<LANG>_COMPILER_LAUNCHER, and both can serve cross builds because the launcher sits in front of the cross compiler. Check effectiveness with ccache -s, and remember that a cache never replaces correct dependency tracking; it only skips work that was already done.

One-command debug loopConfigurecmake--presetBuildcmake--buildDeployrsync toboardStartgdbserverover sshpipeHalt in gdbbreak main↻ Repeat from step 1Configurecmake--presetBuildcmake--buildDeployrsync toboardStartgdbserverover sshpipeHalt in gdbbreak main↻ Repeat from step 1
The one-command loop from source change to a halted debugger
10

Cross-debugging cores, threads and profiling, and common mistakes

5 min read

Core files. A core file, a snapshot of a crashed process's memory and registers written by the kernel, is the field-debugging workhorse. On the board, enable it with ulimit -c unlimited for the process, and decide where it goes with /proc/sys/kernel/core_pattern, described in core(5). On a systemd system the pattern may pipe to systemd-coredump, and coredumpctl lists and extracts them. Cores can be large, so on a flash device write them to tmpfs or a dedicated partition and cap their size. On the host:

bash
gdb-multiarch build/arm64-debug/sensornode core
text
(gdb) set sysroot /home/learner/sysroots/board
(gdb) bt
(gdb) info sharedlibrary
(gdb) thread apply all bt

gdb needs the exact binary and libraries that were running. If you rebuilt since, the addresses no longer match and you see nonsense frames. info sharedlibrary shows whether each library was found with symbols. This is where the build-id archive from the previous section pays off: it is the only way to pair a field core with the right debug file.

Threads. gdbserver reports every thread of the debugged process, and info threads lists them, thread N switches, and thread apply all bt prints every stack, which is the first command to run on a deadlock. By default gdb lets other threads run while you step one; set scheduler-locking on freezes the others, which makes single-stepping predictable at the price of possible deadlock if the stepped thread waits on a frozen one. For threaded programs the host gdb must read libthread_db metadata, which another sysroot library provides, so a wrong or missing sysroot also shows up as missing thread names.

The eSDK and devtool. In the extensible SDK the devtool command gives an application developer an iterative path against the real image without leaving the SDK. devtool add sensornode /home/learner/src/sensornode creates a recipe from your source tree; devtool build sensornode builds it with the full build system; devtool deploy-target sensornode root@board copies the resulting files to the board over SSH; devtool undeploy-target removes them. devtool modify <recipe> extracts the source of an existing recipe into the workspace for patching, and devtool finish turns your changes into a patch in a layer. The eSDK gives you the exact same dependency resolution as the final image build, at the cost of a large installer and more disk space; the plain SDK with CMake is lighter and is the better choice when the application builds standalone.

Profiling across architectures. The rules are asymmetric. perf records on the target against the target's kernel, and the data can be copied to the host and reported with perf report --symfs=<sysroot>, which is usually workable for flat profiles; DWARF-based call-graph unwinding across architectures is the fragile part, so build with frame pointers (-fno-omit-frame-pointer) when you want reliable call graphs. valgrind supports AArch64 and 32-bit ARM Linux natively, but it must run on the target, takes a large amount of memory and time, and does not run under qemu-user; running an x86 valgrind on a foreign binary is not possible. strace and ltrace also run on the target. The AddressSanitizer from earlier lessons works natively on ARM with the cross compiler, and needs its runtime library present in the target's image.

Common mistakes. Four failures account for most lost afternoons in this loop. Each has a symptom, a cause, a diagnosis and a fix:

  • "cannot execute: required file not found" on the board. The binary was built against the host's dynamic loader or a different path than the image provides. Diagnose with file sensornode and readelf -l sensornode | grep interpreter, then compare to the board's loader. Fix: build with the SDK, not the host compiler.
  • "Exec format error" or "cannot execute binary file". The binary is for the wrong architecture, or an AArch64 binary reached a 32-bit userland. Diagnose with file. Fix: check CMAKE_SYSTEM_PROCESSOR and the compiler prefix.
  • Link succeeds with host libraries, program crashes on the board. The find-root modes were BOTH, or pkg-config read host files. Diagnose with cmake --build build --verbose and look for /usr/lib/x86_64-linux-gnu in link lines. Fix: the ONLY modes and PKG_CONFIG_LIBDIR.
  • Breakpoints never hit, or lines look wrong in gdb. The debug file or source does not match the binary on the board, or the optimiser reordered code. Diagnose by comparing build-ids with readelf -n on both copies. Fix: deploy exactly what you built, and use RelWithDebInfo or -Og for development.
Going deeperWhy a missing sysroot can send gdb to the wrong libc

When gdb loads a shared library for a remote target it asks the dynamic linker's link_map list for each library's path on the target, then rewrites that path under the sysroot. With sysroot target: the file is fetched from the board. With a host sysroot, gdb reads <sysroot>/lib/libc.so.6 or the multiarch equivalent. If you point at the host's root by mistake, gdb loads an x86 libc and warns about an architecture mismatch, or silently reads wrong symbols when the architectures are similar, such as 32-bit versus 64-bit ARM. The check is info sharedlibrary, which lists the resolved path for each library; every path should sit under the target sysroot.

Common failures and causesSymptomUsual causeFirst checkrequired filenot foundwrong ELFinterpreterhost gcc usedreadelf -lExec formaterrorwrongarchitecturewrongcompilerprefixfile appHost libs linkedfind modesBOTHpkg-configreads hostverbose linklineBreakpointsnever hitbinary andsymbolsdifferstale debugfilereadelf -nbuild-id
Common failure symptoms in the cross loop and where each one comes from
11

Try it

1 min read

Open the lab for this lesson. Install or unpack an SDK, write a small CMake application with a library target and one pkg-config dependency, and cross-build it with a preset, then confirm with file that the result is for the target. Deploy it to the QEMU target, start gdbserver, attach from the host with the sysroot set, hit a breakpoint and set a watchpoint. Finally write one script or custom target that does all of it in one command. The practice pack has the step-by-step tasks, and none of the answers are in this lesson.

Key takeaways

  • Cross-building means a host compiler emits target machine code, and a sysroot supplies the target headers and libraries.
  • A CMake toolchain file sets CMAKE_SYSTEM_NAME, the compilers, CMAKE_SYSROOT and the find-root modes; presets make it one command.
  • pkg-config needs PKG_CONFIG_LIBDIR and PKG_CONFIG_SYSROOT_DIR, or it silently returns host libraries.
  • Modern CMake attaches usage requirements to targets; PUBLIC, PRIVATE and INTERFACE decide who inherits them.
  • gdbserver runs on the target and host gdb does the symbol work; set sysroot so gdb reads the right libraries.
  • Ship a stripped binary and keep the debug file; the build-id links the two.
  • Reproducible builds need fixed timestamps and path maps, which also makes ccache hits more likely.
  • One script that builds, deploys and starts gdbserver turns a ten-step ritual into a habit.

Reflect

  1. Explain to a colleague, in two sentences: explain the difference between host and target compilation and choose the right one for a given task
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close