Module 2 · Lesson 3 of 30

System calls, errno, strace

How a program asks the Linux kernel for services: system calls, the errno error convention, vDSO, and tracing it all with strace, ltrace and /proc.

  • Beginner to advanced
  • 30 min read
  • 45 min lab
What you will need

gcc and strace on Ubuntu 24.04 (WSL2, VM or container); a Raspberry Pi or other ARM board for the embedded notes

By the end you can 7 objectives
  • Explain what a system call is, why it needs a switch from user mode to kernel mode, and why it costs more than a function call.
  • Distinguish a C library wrapper from the raw syscall() function, and find a call's manual page in section 2 or section 3.
  • Check every return value correctly: tell -1 (error, with errno set) from 0 (end of file) and from a short positive count.
  • Interpret common errno values (ENOENT, EACCES, EPERM, EINTR, EAGAIN, ENOMEM, EIO, EBADF) and write code that saves and reports errno correctly.
  • Run strace with -f, -p, -e trace=, -c, -o, -tt and -y, and read its output line by line, including the = -1 ENOENT form.
  • Analyse the x86-64 and arm64 syscall entry path, the vDSO and vvar mechanism, and ptrace as the engine behind strace.
  • Diagnose a failing or slow program by combining strace filters, fault injection, -k stack traces, ltrace and /proc/PID/syscall.
Read the lesson below; the video will appear here when published.
01

Why system calls matter

2 min read

Every useful thing a Linux program does ends in a request to the kernel. Opening a file, printing a line, starting another program, sleeping for a second, asking for more memory and sending a network packet are all requests. The kernel is the only code allowed to touch hardware and to decide what one process may do to another. A is the only legal way to ask.

This matters to an engineer for three practical reasons. First, almost every bug that involves files, sockets, processes or permissions shows up as a failed system call, and the failure reason is a small integer you must know how to read. Second, system calls are expensive compared with ordinary code, so a program that makes millions of needless calls is slow no matter how clever its algorithms are. Third, you can watch the system calls of any program you can run, without its source code, using strace. On an embedded board with no debugger and no documentation for a vendor binary, that is often the fastest way to learn what the program is really doing.

The idea is old. Early Unix, written at Bell Labs in the early 1970s, already separated a small protected kernel from user programs, and the numbered system call interface was part of that design. Linux, begun by Linus Torvalds in 1991, copied the interface that Unix programs expected, so a C program written for another Unix usually needed few changes. That compatibility is why the names you will meet here, such as open, read, write, fork and exec, are decades old and still the core of the API.

Where system calls sitYour programmain(), your codeC libraryglibc wrappers, stdioSystem call interfacethe only doorwayKernelfiles, processes, memoryHardwaredisk, network, timersrequests go down, results come upYour programmain(), your codeC libraryglibc wrappers, stdioSystem call interfacethe only doorwayKernelfiles, processes, memoryHardwaredisk, network, timersrequests go down, results comeup
Where system calls sit between your program, the C library and the kernel

The picture is a stack. Your code calls a library function, the library makes the system call, the kernel does the privileged work, and hardware sits at the bottom. The rest of this lesson climbs through each layer, then returns to the practical question of how to see it all happen.

02

User mode, kernel mode and the trap

3 min read

A modern CPU can run code at different s. Linux uses two of them. User mode, where your programs run, cannot execute instructions that reconfigure the CPU, cannot access device registers and cannot read kernel memory. Kernel mode, where the kernel runs, can do all of that. The hardware has more levels than two (x86 has rings 0 to 3, ARMv8 has exception levels EL0 to EL3), but Linux uses just the lowest and highest privilege for kernel and user code.

A system call is therefore a controlled change of mode. The program executes a special instruction, called a or exception, and the CPU does three things at once: it raises its privilege, it jumps to an address the kernel configured earlier, and it remembers where to come back. The program cannot choose the destination. That is the entire safety property: the only way into the kernel is through addresses the kernel itself placed in the CPU.

On 64-bit x86 the instruction is syscall. On 64-bit ARM it is svc #0 (supervisor call). On 32-bit ARM it is svc 0 (older documents write swi). On 32-bit x86 older kernels used int 0x80, and later sysenter. Different spelling, same idea. Traps come in three flavours that you should not confuse: a system call is deliberate and synchronous, a page fault or divide-by-zero is an accidental synchronous fault, and an interrupt arrives asynchronously from a device or timer.

1what happens during one call

Step by stepOne system call from user code to result
  1. Prepare: the library puts the syscall number and the arguments in agreed registers.

  2. Trap: the syscall or svc instruction switches the CPU to kernel mode and jumps to the kernel entry point.

  3. Save: the kernel saves the user registers on a kernel stack and switches to its own data.

  4. Dispatch: the kernel uses the number to find the handler in the system call table.

  5. Work: the handler checks arguments, copies data to or from user memory and does the job, possibly sleeping.

  6. Result: the handler returns a value, or a negative error number, in the return register.

  7. Return: the kernel restores the user registers and executes the return-from-trap instruction.

  8. Resume: the program continues in user mode, right after the trap instruction.

One read system callProcessKernelDevicetrap: read(fd, buf,n)request datadata readyreturn count or-errno1. Process → Kerneltrap: read(fd, buf, n)2. Kernel → Devicerequest data3. Device → Kerneldata ready4. Kernel → Processreturn count or -errno
The round trip of one read system call between process and kernel

Why a system call costs more than a function call

A plain function call is a call and a ret plus a few register moves, often a handful of cycles, and the compiler can inline it or optimise across it. A system call cannot be inlined: the CPU must switch privilege, the kernel must save and restore registers, validate every user pointer, run its own code and, on some processors, apply security mitigations that flush or switch state. On many x86 CPUs the mitigations for speculative-execution attacks (Meltdown and Spectre, disclosed in 2018) added measurable cost to every entry and exit, which is why the exact price differs between machines and kernels. The kernel code also pollutes the CPU caches and branch predictors that your program was using.

The consequence is a design rule used everywhere in systems programming: make fewer, larger calls, for example by letting the C library buffer output.

03

Wrappers, syscall numbers and syscall()

5 min read

Look at a line of ordinary C: n = read(fd, buf, 100);. The read here is not the kernel entry point. It is a small function in the C library (glibc on Ubuntu) that places the number for read in the right register, copies its three arguments into the argument registers, executes the trap instruction and converts the result. This function is called a . Wrappers usually map one to one onto a syscall but not always: the library may add argument checking, convert types, or choose between several kernel calls. Modern glibc implements open() with the openat system call, and fork() with clone (or clone3), which is why strace shows names you never typed.

Every system call has a number, and the numbers depend on the architecture. On x86-64, read is 0, write is 1, getpid is 39 and openat is 257. On arm64, read is 63, write is 64, getpid is 172 and openat is 56. Numbers are part of the kernel's stable binary interface to user space (the ), and the kernel never reuses them, so an old binary keeps working on a new kernel. On a Ubuntu development machine, /usr/include/x86_64-linux-gnu/asm/unistd_64.h lists the x86-64 numbers, and in the kernel source arch/x86/entry/syscalls/syscall_64.tbl is the master table.

The raw syscall() function

glibc offers long syscall(long number, ...), declared in <unistd.h>, with the number constants such as SYS_write in <sys/syscall.h>. It performs the trap with the arguments you give and applies the same errno convention as a normal wrapper. Use it when the kernel has a call that glibc has not wrapped yet, or when you want to prove to yourself that no magic happens.

c
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <sys/syscall.h>
#include <unistd.h>

int main(void)
{
    const char msg[] = "hello through syscall()\n";

    long n = syscall(SYS_write, 1, msg, sizeof msg - 1);
    if (n == -1) {
        perror("syscall(SYS_write)");
        return 1;
    }
    printf("wrote %ld bytes\n", n);
    return 0;
}

Build it with gcc -Wall -o rawwrite rawwrite.c, then run it under strace ./rawwrite. The strace line for the write is identical to what an ordinary write(1, msg, len) would produce, because the kernel cannot tell the two paths apart. The raw form gives up type checking and portability: the numbers differ between architectures, and some 32-bit systems split 64-bit arguments across two registers in ways the generic syscall() call leaves to you.

From C call to kernelread(fd,buf, n)your C codeWrapperloadnumber,argumentsTrapsyscall orsvcKernelhandlerdoes theworkResultfixup-errno to-1 anderrnoread(fd,buf, n)your C codeWrapperloadnumber,argumentsTrapsyscall orsvcKernelhandlerdoes theworkResultfixup-errno to-1 anderrno
From a C call to the kernel: wrapper, number, trap and result translation

Feature test macros and manual page sections

Which declarations a header exposes depends on feature test macros, preprocessor symbols you define before the first #include. By default gcc compiles in a GNU mode (-std=gnu17 in gcc 13) that already enables the common POSIX and BSD features. If you use -std=c11 for strict ISO C, many POSIX functions disappear from the headers until you ask for them. Define _POSIX_C_SOURCE as 200809L to request POSIX.1-2008, or _GNU_SOURCE to enable every glibc extension. Defining _GNU_SOURCE has side effects too: for instance, it selects the GNU variant of strerror_r, which returns a char * rather than an int. Each manual page tells you the macro it needs in its Synopsis. man 7 feature_test_macros is the reference.

The manual is split into numbered sections. Section 2 documents system calls as the kernel and C library expose them, and section 3 documents library functions. man 2 read explains the call and its error values, man 3 printf explains a library function that eventually calls write after buffering. man 2 syscalls lists every call with the kernel version that added it. When two sections hold the same name, man 2 name or man 3 name chooses. Memorise the habit: the errors section of a section 2 page is the authoritative list of what can go wrong.

Going deeperKernel side: SYSCALL_DEFINE and the table

In the kernel source, each system call is defined with a macro such as SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf, size_t, count), which you can read in fs/read_write.c. The macro generates the real function, a metadata record for tracing, and wrapping code that sign-extends narrow arguments. The __user annotation marks a pointer that points into user memory and must never be dereferenced directly; the kernel uses copy_from_user() and copy_to_user(), which fail with EFAULT on a bad address instead of crashing the machine. x86-64 numbers come from arch/x86/entry/syscalls/syscall_64.tbl, while arm64 and most newer architectures share the generic list in include/uapi/asm-generic/unistd.h. The kernel documentation page process/adding-syscalls describes how new calls are added.

04

Errors, errno and return values

6 min read

The kernel reports failure by returning a small negative number in the result register: minus the error code. For example, a missing file makes openat return -2. The C library wrapper spots that case and does two things for you: it stores the positive code in a variable named , and it returns -1 to the caller. A kernel result is treated as an error when it falls in the range -4095 to -1, so large positive results such as the address returned by mmap are never mistaken for errors.

errno looks like a global variable, but it is a macro that expands to a per-thread location (in glibc, (*__errno_location())). Each thread has its own copy, so two threads failing at the same moment do not corrupt each other. POSIX requires this. Declare it by including <errno.h>, never by writing extern int errno; yourself, because that declaration breaks thread-local behaviour.

The rules for using errno

Three rules prevent nearly every errno bug.

  1. Check the return value first. Only when a call reports failure is errno meaningful. A successful call is allowed to change errno to any value, so never test errno alone to detect an error.
  2. Read errno immediately. Calls such as printf and malloc can fail internally, or call other functions that set errno, and overwrite it. Copy it into a local variable before doing anything else.
  3. Do not assume a call sets errno to zero on success. If you need a clean slate before a function that signals an error only through errno (like strtol), set errno = 0 yourself before the call.
c
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

int main(void)
{
    int fd = open("/nonexistent/file", O_RDONLY);
    if (fd == -1) {
        int saved = errno;                 /* copy before any other call */
        perror("open");                    /* prints: open: No such file or directory */
        fprintf(stderr, "code %d: %s\n", saved, strerror(saved));
        return 1;
    }
    return 0;
}

perror(msg) prints your message, a colon and the text for the current errno to standard error. strerror(code) returns the text for any code, so you can build your own message. The text depends on the locale, so never parse it; test the numeric constants instead. glibc's printf family also accepts the extension %m, which prints the text for the current errno.

The common error codes

Code Number on Linux Typical meaning
EPERM 1 The operation needs privilege you lack, such as sending a signal to another user's process
ENOENT 2 A file or directory in the path does not exist
EINTR 4 A signal interrupted a blocking call before it finished
EIO 5 The device reported a low-level input or output failure
EBADF 9 The file descriptor is not open, or is open in the wrong mode
EAGAIN 11 The call would block but you asked for non-blocking behaviour, or a resource limit was hit
ENOMEM 12 The kernel could not allocate the memory the call needed
EACCES 13 The file permissions or a directory search permission deny access

On Linux EWOULDBLOCK is defined with the same value as EAGAIN, so one comparison handles both, but portable code should test both names. The easy pair to confuse is EPERM and EACCES: EACCES means the permission bits, or a directory you cannot search, denied the access; EPERM means the operation is restricted to privileged or owning users no matter what the file mode says. The names and numbers come from the kernel headers: asm-generic/errno-base.h holds the first 34 and asm-generic/errno.h the rest, and man 3 errno explains each.

Kernel result versus C resultKernelreturnsC code seesSuccesscount n >= 1nEnd of file00Failure-errno-1 and errno
Two ways of looking at a failed call: the kernel result and what C code sees

Return values are three-way, not two-way

For read and similar calls the result has three meaningful ranges. Minus one is an error and errno says which one. Zero is not an error: it means end of file, or that a peer closed a stream socket or pipe. A positive count is the number of bytes transferred, and it may be smaller than what you requested, which is called a . The next lesson goes deep on short reads and writes; for now the rule is to store the result in a ssize_t and handle all three cases explicitly. Calling the result int and testing if (n) is a bug, because -1 is true.

c
ssize_t n;
do {
    n = read(fd, buf, sizeof buf);
} while (n == -1 && errno == EINTR);       /* retry only on interruption */

if (n == -1) {
    perror("read");                          /* a real failure */
} else if (n == 0) {
    puts("end of file");                     /* orderly end, not an error */
} else {
    /* n bytes arrived, maybe fewer than sizeof buf */
}

Check every call that can fail, including write, close and fsync. A common story in the field is an embedded device that silently loses data because the program ignored the failing write on a full flash filesystem. The error was ENOSPC; the program never looked.

05

Using strace, step by step

6 min read

answers the question "what did this program actually ask the kernel for?". It is the first tool to reach for when a program cannot find a file, hangs, fails with a vague message or runs slowly for no visible reason. Install it with sudo apt install strace on Ubuntu 24.04; Buildroot and Yocto can both add a cross-compiled strace to a board image.

1a basic run

bash
strace cat /etc/hostname

The output goes to standard error. The beginning is process start-up noise from the dynamic loader, and the interesting part is at the end. Typical output (yours will differ), trimmed:

text
execve("/usr/bin/cat", ["cat", "/etc/hostname"], 0x7ffd1c0a2c58 /* 30 vars */) = 0
...
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
...
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
...
openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
...
read(3, "embeddexai\n", 131072)         = 11
write(1, "embeddexai\n", 11)            = 11
read(3, "", 131072)                     = 0
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

2read one line

Every line has the same shape: the call name, its arguments in parentheses, an equals sign and the result. strace decodes constants (O_RDONLY|O_CLOEXEC, AT_FDCWD) and prints strings, cut to 32 characters by default (raise it with -s 128). The result is the raw return value for a success, such as the file descriptor 3 or the byte count 11. A failure prints as a minus one followed by the errno name and its text, for instance = -1 ENOENT (No such file or directory), or = -1 EEXIST (File exists) for an exclusive create that found the file already there. Pointers you passed for the kernel to fill, such as a read buffer, are shown after the call completes.

The read lines in the sample above show the return-value rule from the previous section: the first read returned 11 (a positive count) and the second read returned 0 (end of file). Notice that file descriptor 3 is the first free number after standard input (0), output (1) and error (2).

Anatomy of a strace lineCall nameopenatDecodedargumentsAT_FDCWD,path, flagsEquals signseparatescall andresultResult3 or -1Errno nameENOENT(message)Call nameopenatDecodedargumentsAT_FDCWD,path, flagsEquals signseparatescall andresultResult3 or -1Errno nameENOENT(message)
Anatomy of a strace output line: name, decoded arguments, result and errno

3the options you will use daily

Option Effect
-o file write the trace to a file instead of standard error
-f also follow child processes and threads created by fork, vfork and clone
-p PID attach to a running process instead of starting one
-e trace=openat,read,write show only the named calls; groups such as %file, %network, %process and %signal also work
-e status=failed show only calls that returned an error
-c do not print each call; print a summary table of counts, errors and time when the program ends
-tt prefix each line with a wall-clock time with microseconds; -T appends the time spent inside each call
-y decode file descriptors to their paths, so read(3</etc/hostname>, ...) appears; -yy adds socket details
-s N print up to N characters of each string argument

Combine them: strace -f -tt -y -e trace=%file,read,write -o trace.txt ./myprog. To find why a program cannot find its configuration, filter the failures: strace -e trace=%file -e status=failed ./myprog lists every path it tried and failed to open. Attaching with -p is blocked on Ubuntu, for any process that is not your own descendant, by the Yama ptrace policy (/proc/sys/kernel/yama/ptrace_scope is 1 by default), so use sudo strace -p PID. Press Ctrl+C to detach; the process keeps running.

For the summary, strace -c ./myprog prints a table with one row per syscall and the columns % time, seconds, usecs/call, calls, errors and syscall. By default the time column measures CPU time the kernel spent in the call, not how long the program waited, so a call that sleeps for a long time can look cheap. Add -w to summarise wall-clock latency. Counting calls is usually the first signal: a program that makes ten thousand read calls of one byte each is easy to spot and to fix.

Following forks and pipelines

A pipeline such as cat /etc/hostname | tr a-z A-Z is run by the shell, which creates a pipe and starts each command as a separate process. If you trace only the shell, you see it wait for its children and learn nothing about what they did. The -f option makes strace follow every new process and thread, tagging each line with its process ID: as [pid 1234] on the terminal, or as a plain leading number when you use -o. For one file per process, use -ff -o prefix, which writes prefix.PID files.

bash
strace -f -e trace=execve,openat,read,write,pipe2,dup2,clone,clone3 \
       -o pipeline.txt sh -c 'cat /etc/hostname | tr a-z A-Z'

Ubuntu's sh is dash. You will see the shell create the pipe with pipe2, start children with clone or clone3 (glibc's fork), redirect descriptors with dup2 and replace each child with execve. The write end of the pipe becomes descriptor 1 of the first command and the read end becomes descriptor 0 of the second, so the first command's write(1, ...) and the second command's read(0, ...) carry the same bytes through the kernel's pipe buffer. Matching those two lines by descriptor number and byte count is the core skill of reading a trace.

Mapping descriptors to files

A file descriptor is a small integer that indexes the process's table of open files. By itself read(3, ...) does not say which file it is. Use -y to have strace print the path next to the number, or follow the earlier openat that returned 3. Descriptors are reused after close, so always look for the most recent open that returned that number. For sockets, pipes and devices -y shows forms like 3<pipe:[12345]> or 4<socket:[67890]>, and -yy adds the protocol and addresses for sockets.

Traced pipelinesh -cparent shellcatwrite(1) into pipepipekernel buffertrread(0) from pipesh -cparent shellcatwrite(1) into pipepipekernel buffertrread(0) from pipe
A shell pipeline traced with -f: shell, two children and the pipe between them
06

Inside the kernel entry path

4 min read

On x86-64, the program places the number in rax and the arguments in rdi, rsi, rdx, r10, r8 and r9 (note r10, not rcx, because the syscall instruction overwrites rcx with the return address and r11 with the saved flags). The syscall instruction does not push anything on a stack. It saves the user instruction pointer in rcx, the flags in r11, loads the kernel entry address from a model-specific register (IA32_LSTAR) and raises privilege. The kernel entry code, entry_SYSCALL_64 in arch/x86/entry/entry_64.S, then swaps the segment base to reach per-CPU data, switches to the kernel stack, and on CPUs where the page-table isolation mitigation (KPTI) is active it also switches to the kernel page tables. It builds a pt_regs structure holding the saved user registers and calls do_syscall_64() in arch/x86/entry/common.c. That function bounds-checks the number, calls the matching handler, and stores the return value in rax. An unknown number yields -ENOSYS, which is how a new program on an old kernel learns that a call does not exist yet.

On arm64, the number is in x8, the arguments are in x0 to x5, and the return value comes back in x0. The svc #0 instruction raises an exception from EL0 to EL1; the vector table sends it to el0_svc, which ends in invoke_syscall() in arch/arm64/kernel/syscall.c. On 32-bit ARM with the EABI, the number is in r7 and svc 0 enters the kernel. A board running a 32-bit userland on a 64-bit kernel uses a compatibility table with different numbers, so never assume the numbers from your PC.

x86-64 kernel entrysyscallnumber inraxEntry codeswapstacks,save regsdo_syscall_64boundschecknumberHandlersys_readand othersExit from step 3: unknown number-ENOSYSsysretresult inraxsyscallnumber inraxEntry codeswapstacks,save regsdo_syscall_64boundschecknumberHandlersys_readand othersExit from step 3: unknownnumber-ENOSYSsysretresult inrax
Kernel entry on x86-64: trap, save registers, dispatch, return

Returning to user space

On the way out the kernel runs exit work: it checks for pending signals, a reschedule request and tracing hooks (that is where strace stops the process), then restores the registers and executes sysret on x86-64 or eret on arm64. The wrapper in glibc compares the returned value with the range -4095 to -1, stores the negated value in errno if it is an error, and returns -1. This explains an important edge case: the raw kernel interface returns -EINTR, but the C interface returns -1 with errno == EINTR. If you ever write assembly or use a language runtime that makes raw traps (the Go runtime does), you handle the negative convention yourself.

Going deeperWhy a signal can cancel a sleeping system call

A system call that waits, such as read on a pipe or a terminal, puts the process to sleep inside the kernel in an interruptible state. If a signal with a handler arrives, the kernel wakes the process, sees that the call cannot complete, and arranges the return value -ERESTARTSYS internally. On the way out, if the handler was installed with SA_RESTART, the kernel rewinds the instruction pointer so the call runs again automatically, and the program never notices. Without that flag the call fails with EINTR after the handler runs. Some calls are never restarted whatever the flag says, including poll, epoll_wait, select and nanosleep; man 7 signal lists them. Regular file reads on disk are not interrupted in this way, because a disk wait is uninterruptible (state D in ps), which is also why a stuck NFS mount can leave unkillable processes.

Try the sequence in the widget below, which follows a blocked read when a signal arrives.

A read() that sleeps for data and is hit by a signal runs the handler; without SA_RESTART it then returns -1 with errno EINTR, while with SA_RESTART the kernel reruns the call.

  • states: [{"id": "user", "label": "Running in user mode"}, {"id": "blocked", "label": "Sleeping in read()"}, {"id": "handler", "label": "Signal handler running"}, {"id": "failed", "label": "read() returned -1 EINTR"}, {"id": "done", "label": "read() returned data"}]
  • transitions: [{"from": "user", "to": "blocked", "event": "read() finds no data"}, {"from": "blocked", "to": "done", "event": "data arrives"}, {"from": "blocked", "to": "handler", "event": "signal arrives (no SA_RESTART)"}, {"from": "blocked", "to": "blocked", "event": "signal arrives (SA_RESTART, restarts)"}, {"from": "handler", "to": "failed", "event": "handler returns"}]
07

The vDSO and the vvar page

4 min read

Asking for the time is among the most frequent things programs do, so Linux avoids the system call for it. The kernel maps a small shared object, the , into each process at exec time. Beside it sits the page. You can see both with cat /proc/self/maps, where they appear as [vdso] and [vvar], and ldd /bin/ls lists linux-vdso.so.1, which has no file on disk. The address is also passed to the process in the auxiliary vector as AT_SYSINFO_EHDR, and getauxval() reads it. glibc finds the vDSO symbols at start-up and uses them for clock_gettime, gettimeofday, time and getcpu on x86-64 (named like __vdso_clock_gettime), with __kernel_clock_gettime and friends on arm64.

The mechanism is a lock-free read. The kernel's timekeeping code updates the vvar page after each timer update. The page holds a sequence counter, the time of the last update and the multiplier and shift values that convert hardware counter ticks to nanoseconds. The vDSO function does this:

  1. Read the sequence counter, and retry if it is odd, which means an update is in progress.
  2. Read the hardware counter directly: the time-stamp counter via rdtsc on x86, or the generic timer's virtual count register on arm64.
  3. Compute base time + (ticks since update) * multiplier >> shift.
  4. Read the sequence counter again; if it changed, start over, because the kernel updated the data in the middle.

No privilege change occurs, so the cost is a few dozen cycles of arithmetic instead of a full trap. A direct consequence is that strace shows nothing for clock_gettime calls in a normal run. This is the first thing that surprises people tracing a time-sensitive program.

vDSO versus real syscallvDSO pathReal syscallMode switchnoneuser tokernelData sourcevvar page,counterkerneltimekeepingSeen by stracenoyesUsed whenclocksourceallowsfallback case
clock_gettime through the vDSO versus a real system call

When the fast path is not taken

The vDSO only works when the active clocksource can be read from user mode. On most PC and server hardware the TSC qualifies, but if the kernel falls back to a clocksource such as HPET or the ACPI power-management timer, the vDSO function calls the real system call instead, and strace suddenly shows clock_gettime at huge counts. Some virtual machines and embedded SoCs behave the same way when the timer registers cannot be read from user mode. On a 32-bit ARM board, check with strace before you trust a timestamp-heavy design. The current clocksource is in /sys/devices/system/clocksource/clocksource0/current_clocksource.

Why the vDSO address is randomised

The vDSO is mapped at a randomised address as part of address space layout randomisation (ASLR), the same mechanism that moves the stack and libraries. An attacker who can jump to code of their choosing wants stable snippets of existing code (called gadgets) at known addresses. Before the vDSO, x86-64 used a fixed page called the vsyscall page at 0xffffffffff600000, and because its address never changed it was a favoured target for exploits. Modern kernels emulate or disable it (the vsyscall= boot option chooses), and the randomised vDSO replaced its role. Check ASLR with cat /proc/sys/kernel/randomize_va_space; the usual value is 2.

08

How strace works: ptrace and its cost

3 min read

strace has no special kernel module. It uses the system call, the same interface gdb uses. The tracer starts the program (or attaches with PTRACE_ATTACH/PTRACE_SEIZE), asks the kernel to stop it at every system call with PTRACE_SYSCALL, and then loops.

Step by stepstrace main loop through ptrace
  1. Resume: the tracer calls ptrace with PTRACE_SYSCALL to let the tracee run until its next syscall boundary.

  2. Entry stop: the tracee reaches a syscall entry and the kernel stops it, then wakes the tracer in waitpid.

  3. Read: the tracer reads the registers for the number and arguments, and the tracee memory for strings.

  4. Resume again: the tracer resumes the tracee so the kernel actually performs the call.

  5. Exit stop: the kernel stops the tracee a second time when the call finishes.

  6. Result: the tracer reads the return register, decodes the errno and prints the complete line.

  7. Loop: it resumes the tracee again and waits for the next stop.

ptrace stops per syscallstraceTraceeKernelPTRACE_SYSCALLresumestop at syscallentryread regs and memorysyscall runsstop at syscall exitprint the line1. strace → TraceePTRACE_SYSCALL resume2. Tracee → stracestop at syscall entry3. strace → Kernelread regs and memory4. Tracee → Kernelsyscall runs5. Tracee → stracestop at syscall exit6. strace → straceprint the line
The tracer and tracee alternate at each syscall entry and exit

Each syscall therefore costs two stops, and each stop is a pair of context switches between tracee and tracer plus several ptrace calls to read registers and memory. A traced program that makes many cheap calls can run many times slower, and timing bugs may disappear or shift, which is why strace is a poor tool for latency analysis. Use -c or -e trace= to reduce printing, but note that plain filtering still stops the tracee at every syscall; it only hides lines.

Two more facts matter in practice. Only one tracer may attach to a process at a time, so strace -p on a process that gdb is already controlling fails with "Operation not permitted". The field TracerPid in /proc/PID/status shows the tracer's PID, or 0 for none, which is also the quick way to tell whether a stubborn process is being traced. And a traced program behaves slightly differently: signals are reported to the tracer first, and a program may check for TracerPid to detect and refuse debugging, a trick used by some protection schemes.

Reducing the cost with seccomp-bpf

Modern strace can use the kernel's seccomp filter mechanism to stop the tracee only at the syscalls you asked for. With strace --seccomp-bpf -f -e trace=openat ./prog, strace installs a small BPF program in the tracee that returns an instruction to notify the tracer for openat and to allow everything else at full speed. It works on programs strace starts, not on ones it attaches to, because a filter has to be installed from inside the process. On a slow embedded CPU this can make the difference between a usable trace and a stalled device.

Going deeperWhy seccomp-bpf also matters for sandboxes

The same seccomp mechanism is the base of container and application sandboxes. A filter attached with prctl(PR_SET_SECCOMP) or seccomp() runs at every syscall entry and returns a verdict: allow, return a chosen errno, kill the process or notify a tracer. Container runtimes use it to deny dangerous calls, and a blocked call typically fails with EPERM, so a program that works on your PC may show = -1 EPERM under a container. The Seccomp: line in /proc/PID/status shows 0 for no filter, 1 for strict mode and 2 for a filter. Documentation lives in the kernel at userspace-api/seccomp_filter.

09

Advanced tracing: stack traces, ioctl, injection and ltrace

4 min read

Stack traces with -k

strace -k prints the user-space call stack after each traced syscall, so you can tell which function in which library issued a call. It needs strace built with stack unwinding support, which you can check with strace -V, and the binary should have symbols (build with -g, or install the debug symbols) for readable names. Combine it with a filter to avoid drowning in output: strace -k -e trace=openat -e status=failed ./prog answers "which line of my program, or which library, opens a file that does not exist".

ioctl and fcntl decoding

ioctl is the catch-all call for device-specific operations: ioctl(fd, request, arg). strace decodes the request constant by name where it knows it. A classic example appears in almost every trace of a program that prints to a terminal: ioctl(1, TCGETS, ...). The C library's stdio asks the terminal for its settings to decide whether output should be line-buffered, and when standard output is a pipe or file the call fails with = -1 ENOTTY (Inappropriate ioctl for device), which is the answer "not a terminal". fcntl shows commands such as F_GETFL, F_SETFL with O_NONBLOCK, and F_DUPFD_CLOEXEC, so a non-blocking setup is easy to confirm in a trace.

For a driver of your own, strace may print the request as _IOC(_IOC_READ, 0x45, 0x01, 0x4): a request number is a 32-bit value built from four fields, covered in the widget below. On x86 and ARM the layout is the generic one used by the kernel's include/uapi/asm-generic/ioctl.h: the low 8 bits are the command number, the next 8 the driver type character, then 14 bits of argument size, then 2 bits of direction.

0x80044501 is the Linux input request EVIOCGVERSION: nr 1, type 0x45 (the letter E), size 4 bytes, direction 2 (read from the driver).

  • fields: [{"name": "nr", "lsb": 0, "width": 8}, {"name": "type", "lsb": 8, "width": 8}, {"name": "size", "lsb": 16, "width": 14}, {"name": "dir", "lsb": 30, "width": 2}]
  • value: 0x80044501
  • width: 32

Fault injection

Because error paths are rarely exercised, strace can force them. strace -e inject=openat:error=ENOENT:when=2 ./prog makes the second openat call fail with ENOENT without touching the file system, and -e inject=read:error=EINTR:when=3 makes the third read look interrupted. You can also inject delays with delay_enter and delay_exit to widen race windows. This turns "what happens if this call fails?" from a guess into an experiment.

ltrace, /proc and other tools

ltrace traces calls into shared libraries (such as malloc, strlen and printf) instead of kernel calls; add -S to include system calls and -c for a summary. It works by hooking the dynamic linking tables, so it sees only calls made through shared libraries, not static binaries or calls inside the same library. ltrace is lightly maintained, and on some newer binaries it fails or misses calls; use gdb breakpoints, uftrace or bpftrace as alternatives. On a target board with no strace, the perf trace command (if the kernel has perf events) gives a lower-overhead system call trace, and the kernel's own tracepoints under /sys/kernel/tracing need no ptrace at all.

For a process that is already running and stuck, read the kernel's view directly. /proc/PID/syscall shows the syscall number, six arguments in hex, then the stack pointer and program counter, if the process is blocked in a call, running if it is on a CPU, or -1 followed by the stack pointer and program counter if it is not in a system call. /proc/PID/status shows State: (S for sleeping, D for uninterruptible), TracerPid:, Seccomp: and the context-switch counters. /proc/PID/wchan names the kernel function the process sleeps in. Reading these needs the same ptrace-access permission as attaching, so another user's process requires root.

bash
sleep 300 &
cat /proc/$!/syscall           # on x86-64 typically starts with 230, clock_nanosleep
grep -E 'State|TracerPid|Seccomp' /proc/$!/status
kill $!

At the raw kernel interface a failure returns -errno: -2 (ENOENT) is the 64-bit pattern 0xFFFFFFFFFFFFFFFE, inside the error range -4095 to -1, so the wrapper stores 2 in errno and returns -1.

  • value: -2
  • width: 64
Choosing a tracing toolstraceltraceperf traceTracessystem callslibrary callssystem callsMethodptrace stopsPLT hookskernel eventsOverheadhighhighlower
Choosing between strace, ltrace, perf trace and /proc for a question
10

Common mistakes and how to diagnose them

2 min read

Here are the failures that show up repeatedly in real code, with how to recognise and fix each.

Wrong message from perror. Symptom: an error message names an unrelated reason, such as "Inappropriate ioctl for device" after a file open failure. Cause: a call between the failure and the perror (often printf or a library call) overwrote errno. Diagnose by tracing and comparing the real result in strace with the message. Fix: copy errno into a local variable immediately after the failing call.

Intermittent EINTR failures. Symptom: a program works for hours, then a read or accept fails with "Interrupted system call" after you add a timer or a signal handler, or after someone sends SIGUSR1. Cause: a signal arrived while the call was blocked, and the handler was installed without SA_RESTART, or the call is one that is never restarted. Diagnose: strace shows --- SIGALRM {...} --- followed by read(...) = -1 EINTR. Fix: loop on EINTR as in the earlier example, or install handlers with SA_RESTART when appropriate, and prefer poll with a signal-safe wakeup in event loops.

A busy loop on EAGAIN. Symptom: one CPU core is pinned at 100 percent and the device is idle. Cause: a non-blocking descriptor is read in a tight loop and keeps returning EAGAIN. Diagnose: strace -c -p PID for a few seconds shows an enormous count of a single call with the same number of errors. Fix: wait for readiness with poll, select or epoll, then read.

Ignoring errors from write and close. Symptom: truncated or missing files on a full or failing flash filesystem. Cause: unchecked return values. Diagnose: strace -e status=failed shows ENOSPC or EIO. Fix: check every call; for close, report the error but do not retry it on Linux, because the descriptor is already released and a retry might close a descriptor that another thread just reused.

strace itself hides the bug. Symptom: the race or timeout disappears under strace. Cause: ptrace stops change the timing. Fix: use --seccomp-bpf with a narrow filter, or perf trace, or kernel tracepoints.

Failing program workflowProgramfailsvaguemessagestrace -f-ocaptureeverythingFilterfailuresstatus=failedFind errnoENOENT,EACCES, ...Exit from step 2: attach refusedcheck ptrace_scopeFix causepath, mode,handlingProgramfailsvaguemessagestrace -f-ocaptureeverythingExit from step 2: attachrefusedcheck ptrace_scopeFilterfailuresstatus=failedFind errnoENOENT,EACCES, ...Fix causepath, mode,handling
A decision path from a failing program to the call that explains it
11

Try it

1 min read

Open the lab for this lesson and run strace -c on a real program to rank its calls, trace a shell pipeline across forks with -f and map each descriptor to its file, and write a short program that triggers EINTR and EAGAIN and prove both in the trace. Then open the practice pack: it has warm-up, core, challenge and expert exercises, quiz questions and interview-style questions on system calls, errno and tracing. Use man 2 syscalls and the section 2 page of each call you meet as your reference.

Key takeaways

  • A system call is a controlled jump into the kernel; the CPU mode switch, register handling and mitigations make it far costlier than a function call.
  • glibc wrappers turn the kernel's negative return value into -1 plus errno; syscall() lets you call by number without a wrapper.
  • errno is thread-local, meaningful only right after a call that reported failure, and is overwritten by almost any later call.
  • For read-like calls, -1 means error, 0 means end of file, and a positive value may be less than you asked for.
  • strace shows every syscall with decoded arguments and the result, including = -1 ENOENT (No such file or directory).
  • clock_gettime usually never traps: the vDSO reads a kernel-maintained data page in user space, so strace does not see it.
  • strace works through ptrace, which stops the traced program twice per syscall; filtering and --seccomp-bpf cut that cost.
  • A blocked process shows its current syscall in /proc/PID/syscall, and TracerPid in /proc/PID/status shows who is tracing it.

Reflect

  1. Explain to a colleague, in two sentences: explain what a system call is, why it needs a switch from user mode to kernel mode, and why it costs more than a function call.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close