Module 2 · Lesson 13 of 30

Threads, mutexes, condition variables

Build safe pthread lifetimes and bounded queues; understand mutexes, condition variables, futexes, thread storage, cancellation and priority.

  • Beginner to advanced
  • 31 min read
  • 60 min lab
What you will need

Ubuntu 24.04 in a VM, WSL2 or native Linux; GCC, GDB and a supported ThreadSanitizer runtime; optional ARM Linux board

By the end you can 6 objectives
  • Distinguish process resources, per-thread state, Linux TIDs and pthread handles.
  • Create and join workers with safe argument and result lifetimes.
  • Explain data races and establish happens-before relationships with mutexes.
  • Implement bounded producer-consumer operations with predicate loops and coordinated shutdown.
  • Apply thread-local storage, one-time initialization and appropriate thread attributes.
  • Diagnose races and deadlocks, and evaluate robust recovery, cancellation and priority inheritance.
Read the lesson below; the video will appear here when published.
01

One process, several workers

3 min read

Imagine a board sampling a sensor while sending earlier samples over a slow network. A lets one worker sample while another waits for the network.

Concurrency means overlapping progress; parallelism means simultaneous execution on different CPU execution units. A single-core board supports concurrency without extra computing power.

The previous lesson tracked allocation ownership. Thread-safe allocation does not make allocated objects safe to share. Two valid pointers can still reach the same unprotected list.

Processes normally have separate virtual address spaces. After fork(), ordinary private mappings do not become shared writable objects. Threads share an address space, including global variables, heap allocations and all their stacks. A pointer to a live stack object can therefore reach another thread. Its lifetime still ends when its declaring function returns.

Each thread has its own stack, registers and signal mask. , or TLS, holds per-thread values. C libraries provide per-thread errno. Signal dispositions, such as installed handlers, remain shared.

Closing a descriptor affects all threads using its number. Another open can reuse it. A worker's local integer does not own an independent file.

Linux exposes several identities:

Identity How to obtain it Meaning
Process ID getpid() Shared process identity for these threads
Linux thread ID gettid() Kernel task identity; different for each thread
POSIX thread handle pthread_self() Opaque pthread_t used by pthread interfaces

A is not a pthread_t. Compare pthread handles with pthread_equal(a, b), not integer assumptions. Handles can be reused after their lifetime ends.

POSIX threads, or pthreads, are standard interfaces in <pthread.h>, implemented by Linux libraries above kernel task facilities. See pthreads(7) for the resource split.

Shared resources and per-thread stateProcess-widePer threadMemorySharedaddress spaceOwn stackregionFilesShareddescriptortableNo privatedescriptorcopyExecutionSharedprogram codeOwn registersStored valuesGlobals andheapTLS instancesIdentitySame getpidresultDistinctLinux TIDSignalsShareddispositionsOwn signalmask
Threads share process resources while each keeps its own execution state.
02

Create workers and finish their lifetimes

3 min read

1Prepare the argument before creation

pthread_create() stores a handle, takes optional attributes, and starts a function with one void * argument. Either thread may run first. Initialize arguments before creation.

Save this complete example as lifetime.c. Later fragments reuse must(), which terminates this teaching program on a pthread error. Reusable libraries should return errors instead.

c
#define _POSIX_C_SOURCE 200809L
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static void must(int rc, const char *operation)
{
    if (rc != 0) {
        fprintf(stderr, "%s: %s\n", operation, strerror(rc));
        exit(EXIT_FAILURE);
    }
}

struct work {
    unsigned input;
    unsigned result;
};

static void *calculate(void *argument)
{
    struct work *work = argument;
    work->result = work->input * work->input;
    return work;
}

int main(void)
{
    struct work tasks[2] = {{.input = 3}, {.input = 5}};
    pthread_t workers[2];
    for (size_t i = 0; i < 2; ++i)
        must(pthread_create(&workers[i], NULL, calculate, &tasks[i]),
             "pthread_create");
    for (size_t i = 0; i < 2; ++i) {
        void *result;
        must(pthread_join(workers[i], &result), "pthread_join");
        struct work *completed = result;
        printf("%u squared is %u\n", completed->input, completed->result);
    }
    return EXIT_SUCCESS;
}

Build and run it with the following commands. -pthread selects compiler and linker support for threads; retain it with your cross compiler too.

bash
gcc -std=c11 -Wall -Wextra -Wpedantic -g -pthread lifetime.c -o lifetime
./lifetime

The program prints inputs and squares after joining. The task array stays alive; workers write distinct objects that main reads only after joining.

2Collect results exactly once

pthread_join(t, &result) waits for termination, obtains the returned pointer and makes earlier worker writes visible. Passing NULL discards the result. Join each joinable worker once; concurrent joins are invalid.

Returning from a worker's start function terminates that thread. pthread_exit(value) can terminate it from a deeper call. Neither turns a pointer into copied data. Never return a pointer to a worker's automatic local variable. Returning from main() or calling exit() terminates the process and its other threads.

3Choose joinable or detached ownership

pthread_detach(t) arranges automatic reclamation of thread resources after termination. It neither stops the worker nor frees application allocations. Detached workers cannot be joined and need another completion scheme for safe shared-object shutdown.

Step by stepA joinable worker's lifetime
  1. Prepare: Initialize a task object whose lifetime spans the worker's use.

  2. Create: Start the worker and save its handle after successful creation.

  3. Execute: The worker uses its argument while the creator can do other work.

  4. Finish: The worker returns a pointer to an object that is still alive.

  5. Join: One owner waits and collects completion and the result pointer.

  6. Reclaim: Release task storage only after its final user has finished.

Most pthread functions return error numbers without setting errno; perror() alone misdiagnoses creation failures. If aborting startup, stop and join already-created workers. This example exits the process instead. See pthread_create(3).

Who reclaims a worker's resources?Joinable workerFinished, joinableDetached workerResources reclaimedTransitionsWorker finishesJoinable workerFinished, joinablepthread_detachJoinable workerDetached workerpthread_joinFinished, joinableResources reclaimedpthread_detachFinished, joinableResources reclaimedWorker finishesDetached workerResources reclaimedJoinable workerFinished, joinableDetached workerResources reclaimedTransitionsWorker finishesJoinableworkerFinished,joinablepthread_detachJoinableworkerDetachedworkerpthread_joinFinished,joinableResourcesreclaimedpthread_detachFinished,joinableResourcesreclaimedWorker finishesDetachedworkerResourcesreclaimed
Joinable and detached workers have different resource-reclamation paths.
03

Protect shared rules with mutexes

3 min read

Two workers incrementing counter can read the same value and overwrite each other's update. This explains one failure, but C gives a stronger warning.

A makes a C11 program's behaviour undefined. Non-atomic means an ordinary access without the atomic guarantees discussed later. A compiler need not preserve the imagined load-add-store sequence. A correct-looking final total proves nothing.

A race condition means correctness depends on timing. Separately locked checks and updates can race without a C data race: two callers observe one free slot, then both claim it.

A should preserve an invariant. An invariant is a rule that must hold at the boundaries of each protected operation. For a queue, its count must agree with its occupied slots.

A provides exclusion and memory ordering. This fragment uses must() from the previous example:

c
static pthread_mutex_t count_lock = PTHREAD_MUTEX_INITIALIZER;
static unsigned long counter;

static void record_sample(void)
{
    must(pthread_mutex_lock(&count_lock), "lock counter");
    ++counter;
    must(pthread_mutex_unlock(&count_lock), "unlock counter");
}

Concurrent readers must also hold count_lock; reading after joining all writers is safe too. An unlocked monitoring reader still races. volatile changes certain compiler access rules, without supplying exclusion or thread synchronization.

Use PTHREAD_MUTEX_INITIALIZER when default attributes suit a declared object. For runtime construction, call pthread_mutex_init(&m, &attributes) or pass NULL for defaults. Check success before exposing the object. Do not initialize it twice, copy it with assignment or memcpy, or move its containing allocation with realloc() while it is live.

pthread_mutex_lock() waits until it acquires ownership. pthread_mutex_trylock() returns immediately; EBUSY means an ordinary mutex is unavailable. Handle this result as contention, not a fatal error. Repeated trylock loops waste CPU and can delay the owner on small boards.

Explicit type Relock by its owner Typical use
PTHREAD_MUTEX_NORMAL Deadlocks Simple code with proven ownership
PTHREAD_MUTEX_ERRORCHECK Returns EDEADLK Finding ownership mistakes
PTHREAD_MUTEX_RECURSIVE Increments an ownership count Interfaces deliberately requiring recursive entry

Choose a type using pthread_mutexattr_init(), pthread_mutexattr_settype(), then pthread_mutex_init(). Destroy attributes afterward. Recursive owners must match every lock with an unlock. PTHREAD_MUTEX_DEFAULT does not promise ERRORCHECK behaviour; consult the POSIX mutex rules.

Only the owner may unlock. Non-owner unlocking of a non-robust NORMAL mutex is undefined behaviour; ERRORCHECK reports EPERM.

Keep slow I/O and callbacks outside locks. Call pthread_mutex_destroy() only when unlocked, with no waiter or future user. Destruction does not stop threads.

Protect the complete operationAcquiremutexCheck stateRead allrelatedfieldsUpdateinvariantKeep countand slotsconsistentReleasemutexAcquiremutexCheck stateRead allrelatedfieldsUpdateinvariantKeep countand slotsconsistentReleasemutex
The same mutex encloses every read and update of related shared state.
04

Wait for a condition without losing changes

4 min read

An empty queue needs efficient waiting. Repeated checks waste CPU and board power; guessed sleep intervals add delay without proving correctness.

A gives waiting a precise meaning. For a consumer, it can be count > 0 || closed. A connects that check to efficient waiting.

1Test while holding the mutex

Protect reading and changing the predicate with the same mutex. An unlocked check lets a producer signal before you wait, leaving you asleep despite available work: a lost wakeup.

2Release and wait as one operation

Call pthread_cond_wait(&changed, &lock) while owning lock. It releases that mutex and begins waiting atomically with respect to the other thread's mutex-protected change and notification. On successful return it owns the mutex again. This closes the gap between checking and waiting.

3Recheck after waking

Use while, never a single if. A spurious wakeup is a return without a matching useful notification. Also, another consumer may take the item before you reacquire the mutex. Both cases require the same predicate loop.

pthread_cond_signal() unblocks at least one waiter if any exist; do not rely on exactly one return or any chosen waiter. pthread_cond_broadcast() unblocks all current waiters. Neither hands over the mutex. Awakened threads compete to acquire it before returning from their waits.

An empty, open queue makes the consumer wait with the mutex released; after any wakeup it reacquires the mutex and rechecks count > 0 || closed, waiting again if the queue is still empty and open.

  • states: [{"id": "check", "label": "Check count > 0 || closed; mutex held"}, {"id": "waiting", "label": "Waiting on condition; mutex released"}, {"id": "reacquire", "label": "Awakened; must reacquire mutex"}, {"id": "proceed", "label": "Predicate true; handle item or closure under mutex"}]
  • transitions: [{"from": "check", "to": "waiting", "event": "Empty and open: cond_wait atomically releases and waits"}, {"from": "waiting", "to": "reacquire", "event": "Signal, broadcast or spurious wakeup"}, {"from": "reacquire", "to": "check", "event": "Acquire mutex before returning from cond_wait"}, {"from": "check", "to": "proceed", "event": "Item available or queue closed"}]

Update the predicate and notify while holding its mutex. POSIX permits unlocked signaling, but still requires a sound predicate protocol. Notifications sent without waiters are not stored; shared state remembers work.

Step by stepAn empty queue becomes readable
  1. Check: The consumer locks and finds the queue empty and open.

  2. Wait: Condition waiting releases the mutex and registers the wait without a notification gap.

  3. Publish: The producer acquires the mutex, inserts an item and changes the count.

  4. Notify: The producer signals the consumer condition, then unlocks.

  5. Reacquire: An awakened consumer obtains the same mutex again.

  6. Recheck: The consumer checks the predicate and either consumes or waits again.

Initialize with PTHREAD_COND_INITIALIZER or pthread_cond_init(). Concurrent waits on one condition variable must use the same mutex while that binding exists. Destroy it only after no thread can wait or signal again. Avoid recursive mutexes here: waiting may fail to release the mutex fully when its recursion count exceeds one.

For a deadline, pthread_cond_timedwait() takes an absolute time, not a duration. Its default clock is CLOCK_REALTIME, which can jump. A condition attribute configured using pthread_condattr_setclock(..., CLOCK_MONOTONIC) permits deadlines on a clock that does not jump with wall-clock changes. Even after ETIMEDOUT, check the predicate under the reacquired mutex; timeout and publication can race. See condition-wait semantics.

Wait, notify and recheckConsumerProducerConditionLock; find emptyWait; release mutexLock; insert itemSignalUnlockWakeReacquire; recheck1. Consumer → ConsumerLock; find empty2. Consumer → ConditionWait; release mutex3. Producer → ProducerLock; insert item4. Producer → ConditionSignal5. Producer → ProducerUnlock6. Condition → ConsumerWake7. Consumer → ConsumerReacquire; recheck
Waiting releases the mutex atomically with entering the wait and reacquires it before rechecking.
05

Build bounded queue operations

2 min read

A bounded buffer limits memory and creates backpressure: producers slow down when consumers cannot keep up. Choose blocking, dropping samples or reporting overload for your board. This example blocks.

This queue module stores integers by value. Include <stdbool.h>, <stddef.h> and <pthread.h>, reuse must(), and define _POSIX_C_SOURCE=200809L before headers. Integrate it into your lab driver.

c
#define QUEUE_CAPACITY 8
struct queue {
    pthread_mutex_t lock;
    pthread_cond_t readable;
    pthread_cond_t writable;
    int slots[QUEUE_CAPACITY];
    size_t head, tail, count;
    bool closed;
};

static struct queue samples = {
    .lock = PTHREAD_MUTEX_INITIALIZER,
    .readable = PTHREAD_COND_INITIALIZER,
    .writable = PTHREAD_COND_INITIALIZER
};

static bool queue_put(struct queue *q, int value)
{
    must(pthread_mutex_lock(&q->lock), "put lock");
    while (q->count == QUEUE_CAPACITY && !q->closed)
        must(pthread_cond_wait(&q->writable, &q->lock), "wait space");
    if (q->closed) {
        must(pthread_mutex_unlock(&q->lock), "put closed unlock");
        return false;
    }
    q->slots[q->tail] = value;
    q->tail = (q->tail + 1) % QUEUE_CAPACITY;
    ++q->count;
    must(pthread_cond_signal(&q->readable), "signal item");
    must(pthread_mutex_unlock(&q->lock), "put unlock");
    return true;
}

static bool queue_get(struct queue *q, int *value)
{
    must(pthread_mutex_lock(&q->lock), "get lock");
    while (q->count == 0 && !q->closed)
        must(pthread_cond_wait(&q->readable, &q->lock), "wait item");
    if (q->count == 0) {
        must(pthread_mutex_unlock(&q->lock), "get drained unlock");
        return false;
    }
    *value = q->slots[q->head];
    q->head = (q->head + 1) % QUEUE_CAPACITY;
    --q->count;
    must(pthread_cond_signal(&q->writable), "signal space");
    must(pthread_mutex_unlock(&q->lock), "get unlock");
    return true;
}

static void queue_close(struct queue *q)
{
    must(pthread_mutex_lock(&q->lock), "close lock");
    q->closed = true;
    must(pthread_cond_broadcast(&q->readable), "wake consumers");
    must(pthread_cond_broadcast(&q->writable), "wake producers");
    must(pthread_mutex_unlock(&q->lock), "close unlock");
}

Zero-initialized fields start with an empty, open queue. The invariant is 0 <= count <= QUEUE_CAPACITY; both indices stay within the array. All queue fields are protected by lock. The modulus wraps an index back to zero. These fields need no atomics because every shared access uses the mutex.

queue_get() needs writable output storage reserved for its caller. Success removes one item; false means closed and drained. Consumers finish queued work after closure; producers reject new work.

1Separate coordination from processing

Start consumers before filling a queue that could block producers. Prepare and process items outside the queue lock. A slow network write inside queue_get() would prevent unrelated puts.

2Give shutdown an owner

For normal completion, the coordinator joins all producers, closes the queue, then joins all consumers. An individual producer must not close it while other producers still have work. For an early stop, the coordinator can close first, causing blocked puts to return false. Broadcast both conditions, since either group can be asleep.

3Release objects after the final join

After all users finish, destroy both conditions and the mutex. Never free a queue with possible future users. This module uses cooperative shutdown; cancellation needs the cleanup covered later.

For pointer items, define ownership on every outcome. Failure can leave ownership with the producer; successful put and get transfer it through the queue to the consumer. Account for discarded items. A thread pool adds a callback and argument while retaining these shutdown rules.

The bounded queue's shared objectsQueueMutexProtects all ring stateRing statehead, tail, count, closedItem slotsFixed capacity; integer valuesReadable conditionItems available or queue closedWritable conditionSpace available or queue closedQueueMutexProtects all ring stateRing statehead, tail, count, closedItem slotsFixed capacity; integervaluesReadable conditionItems available or queueclosedWritable conditionSpace available or queueclosed
One mutex protects ring state, while separate conditions represent readable items and writable space.
06

Keep library state local and initialize it once

3 min read

A function is thread-safe when concurrent calls preserve its documented behaviour. Reentrancy means a function can be entered again before an earlier call finishes without corrupting its state. A function using an internal mutex may be thread-safe but deadlock if a signal handler interrupts it and calls it again.

Use the specific async-signal-safe list for signal handlers; thread-safe does not mean safe there. Neither pthread_mutex_lock() nor condition signaling belongs in a portable signal handler. The signal lesson's minimal handlers or dedicated signal-waiting threads remain the appropriate patterns.

strtok_r() keeps parsing state in a caller-supplied pointer. Independent parses need separate state and input buffers. Sharing an input buffer still needs coordination.

C11 provides _Thread_local for per-thread instances. GCC's __thread is a compiler extension with a similar purpose. For example, this declaration gives each worker a separate counter:

c
static _Thread_local unsigned samples_seen;

The object persists until thread termination. It does not free allocations whose pointers it stores. Pointers into TLS also become invalid when the owning thread exits.

POSIX thread-specific data provides keys instead of named compiler TLS objects. pthread_key_create() creates one process-wide key. pthread_setspecific() associates a pointer with that key for the calling thread; pthread_getspecific() retrieves that thread's pointer. A destructor can release it at thread exit.

This fragment allocates one scratch object lazily per thread. Add <errno.h> to the earlier includes. Failure to set the pointer frees the allocation immediately:

c
struct scratch { char text[128]; };
static pthread_key_t scratch_key;
static pthread_once_t scratch_once = PTHREAD_ONCE_INIT;
static int scratch_key_error;

static void make_scratch_key(void)
{
    scratch_key_error = pthread_key_create(&scratch_key, free);
}

static struct scratch *thread_scratch(void)
{
    must(pthread_once(&scratch_once, make_scratch_key), "scratch once");
    must(scratch_key_error, "scratch key");
    struct scratch *s = pthread_getspecific(scratch_key);
    if (s == NULL) {
        s = calloc(1, sizeof *s);
        if (s == NULL)
            must(ENOMEM, "scratch allocation");
        int rc = pthread_setspecific(scratch_key, s);
        if (rc != 0) {
            free(s);
            must(rc, "scratch association");
        }
    }
    return s;
}

pthread_once() completes successful initialization before callers return through that control. Its void initializer records failure separately. An unlocked static bool initialized would race. Never recursively invoke the same once control.

Going deeperStorage cleanup has boundaries

Replacing a thread-specific pointer with pthread_setspecific() does not destroy the old value. Deleting a key does not run destructors either. At thread exit, non-null associations with destructors are processed; destructors can lead to bounded additional passes if they install new values. Avoid relying on destructor ordering between keys. Process exit does not guarantee another thread's destructors will run. Join workers and release their resources explicitly when shutdown matters. See thread-specific storage.

TLS access also has a binary cost. The compiler and linker choose an access method based on whether storage lives in the executable or a shared library. A shared library can require a runtime address lookup. Do not force a TLS model copied from desktop code into a loadable board library without checking its loading requirements.

A shared key, separate valuesProcessOnce controlInitializes shared scratch keyWorker ATLS counterA's samples_seen instanceScratch pointerA's value under shared keyWorker BTLS counterB's samples_seen instanceScratch pointerB's value under shared keyProcessOnce controlInitializes shared scratchkeyWorker ATLS counterA's samples_seen instanceScratch pointerA's value under sharedkeyWorker BTLS counterB's samples_seen instanceScratch pointerB's value under sharedkey
Named TLS and key-based storage give separate worker values, while a once control initializes the shared key.
07

Follow ordering from C to the futex

3 min read

A producer writes a slot, then unlocks. A consumer locks the same mutex before reading. That protocol makes earlier writes visible without volatile.

The C11 memory model describes which cross-thread observations programs may rely on. combines order within a thread with synchronization between threads. Conflicting ordinary accesses need this ordering to avoid a data race.

Unlocking a mutex has release semantics; acquiring it has acquire semantics. Operations before a release become ordered before operations after the matching later acquisition. This is not a requirement to flush every cache to RAM. The compiler and processor implement the necessary ordering for their architecture.

Creation publishes initialized arguments; joining orders completed worker actions before subsequent reads. Neither protects accesses occurring concurrently between creation and completion.

An atomic object supports indivisible accesses under C's rules. C11 declares atomic types and operations in <stdatomic.h>. A release store to an atomic ready flag can publish preceding ordinary data writes when an acquire load reads that store's value. A relaxed atomic counter can safely count independent events but does not by itself publish unrelated payload bytes.

Do not infer that an atomic type is lock-free on every board. Lock-free means operations do not require a blocking lock and provide system-wide progress; it does not promise every individual thread completes promptly. atomic_is_lock_free() helps check an object's implementation. The next lesson builds the full publication and reuse rules for ring slots.

A connects user-space lock bookkeeping to kernel waiting. An ordinary uncontended pthread mutex usually succeeds through user-space atomic operations. It does not need a system call for every lock.

When contention requires sleep, a library can request FUTEX_WAIT. The kernel compares the word with an expected value and blocks only if they match, as one atomic compare-and-block operation. If the value changed already, waiting fails with EAGAIN, allowing the library to retry instead of sleeping after a missed change.

The owner updates lock state before requesting FUTEX_WAKE when needed. Wake makes selected waiters runnable; it does not transfer ownership or guarantee immediate execution. Waiters recheck user-space state. These are operation names; futex_wait and futex_wake are common helper names, not portable pthread APIs. See futex(2).

Going deeperRead implementation code without making it an ABI

In glibc, useful entry points include nptl/pthread_mutex_lock.c, nptl/pthread_cond_wait.c and nptl/pthread_create.c. A condition variable needs bookkeeping for waiters, notification eligibility and cancellation; it is not simply the queue count used as a futex word. Inspect your exact library source when debugging. musl provides pthread interfaces through its own implementation. Neither library's private mutex layout is a portable application interface. Never edit its fields or issue futex calls against them yourself.

ARM may permit observations an x86 test never reveals. Pthreads and C atomics supply the required architecture instructions. Aligned ordinary loads, delays and CPU affinity cannot replace ordering.

An ordinary mutex's fast and slow pathsAtomicacquisitionAttempt inuser spaceContentionremainsLibraryprepares towaitFUTEX_WAITCompareexpectedvalue;block ifequalOwnerreleasesPublishunlockedstateExit from step 1: SuccessOwn mutexFUTEX_WAKEMake awaiterrunnableRetryacquisitionWake doesnot grantownershipAtomicacquisitionAttempt inuser spaceContentionremainsLibraryprepares towaitExit from step 1: SuccessOwn mutexFUTEX_WAITCompareexpectedvalue;block ifequalOwnerreleasesPublishunlockedstateFUTEX_WAKEMake awaiterrunnableRetryacquisitionWake doesnot grantownership
Atomic user-space acquisition is the fast path, while contention can reach futex compare-and-block and wake.
08

Budget stacks and understand scheduling

4 min read

Initialize pthread_attr_t with pthread_attr_init(), set properties, pass it to pthread_create(), then call pthread_attr_destroy(). Destruction releases attributes, not the created thread.

pthread_attr_setstacksize() requests a stack size in bytes. Requirements include the platform's minimum and possible alignment constraints. Check the return value instead of assuming a chosen number works everywhere. The stack holds nested call frames, automatic arrays and library calls, so your worker function's local variables are only part of the budget.

Defaults differ across libraries, architectures and startup resource limits. Distinguish reserved address space from resident pages. Compare reservations, resident use and worst-case call depth. Guard regions catch some overflows without making undersized stacks safe.

This fragment changes the earlier example's creation settings. The size is an illustrative request, not a safe minimum for arbitrary applications:

c
pthread_attr_t attr;
must(pthread_attr_init(&attr), "attr init");
size_t requested_stack = 256u * 1024u;
must(pthread_attr_setstacksize(&attr, requested_stack), "stack size");
must(pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE),
     "detach state");
/* Use &attr instead of NULL in each pthread_create call. */
must(pthread_attr_destroy(&attr), "attr destroy");

Create before destroying attributes. Supplying your own stack adds size, lifetime and overflow-protection responsibilities. Prefer library-managed stacks; see stack-size attributes.

Linux schedules each pthread as a kernel task. In glibc this uses the Native POSIX Threads Library, or NPTL, with one kernel task per application thread. Kernel creation uses the clone family with flags sharing resources; exact calls depend on library and kernel support. The task's kernel stack is separate from the user stack you sized.

A scheduling policy decides how runnable threads receive CPU time. Ordinary SCHED_OTHER threads use a static scheduling priority of zero; nice values influence fair scheduling. SCHED_FIFO and SCHED_RR are real-time policies with higher-priority selection rules. FIFO runs an eligible thread until it blocks, yields or is preempted by higher priority. RR also time-slices peers at the same priority.

Inspect or change settings with pthread_getschedparam() and pthread_setschedparam(). Query limits using sched_get_priority_min() and sched_get_priority_max(). Real-time creation can fail with EPERM despite successful attribute setters, because privileges or resource limits are insufficient.

To apply scheduling values from creation attributes, set PTHREAD_EXPLICIT_SCHED with pthread_attr_setinheritsched(). The default is inheritance from the creator, which ignores supplied policy and priority attributes. See inheritance rules.

Priority inversion lets a medium-priority task delay a low-priority lock owner needed by a high-priority waiter. A priority-inheritance mutex temporarily boosts the owner to help it release the lock.

Configure a fresh mutex by setting pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT) before pthread_mutex_init(). Check support and every error. Linux uses priority-inheritance futex operations for this path; ordinary WAIT/WAKE alone cannot supply it. Inheritance can propagate through lock dependencies, but it does not repair deadlock or bound slow I/O inside the lock. Read the protocol rules before making timing claims.

Match the target environment

BusyBox supplies small command-line tools; it does not determine pthread behaviour. Your target C library and kernel do. Cross-compile with the board SDK, its headers and matching libraries. TSan runtime availability, real-time permissions and diagnostic commands can differ from your development PC.

Helping the lock owner runLow ownerHigh waiterSchedulerAcquire PI mutexBlock on mutexBoost effective priorityFinish protectedworkRelease mutexRecompute effective priorityWaiter can proceed1. Low owner → Low ownerAcquire PI mutex2. High waiter → SchedulerBlock on mutex3. Scheduler → Low ownerBoost effective priority4. Low owner → Low ownerFinish protected work5. Low owner → SchedulerRelease mutex6. Scheduler → Low ownerRecompute effective priority7. Scheduler → High waiterWaiter can proceed
Priority inheritance helps a low-priority lock owner run when a high-priority waiter depends on it.
09

Handle owner death and cancellation deliberately

4 min read

A robust mutex reports owner death to a later acquirer. Robustness is independent of NORMAL, ERRORCHECK or RECURSIVE type. Non-robust mutexes do not recover automatically. Set pthread_mutexattr_setrobust(&a, PTHREAD_MUTEX_ROBUST) on initialized attributes before constructing the mutex.

If locking returns EOWNERDEAD, the caller has acquired the mutex and must treat its protected state as inconsistent. Inspect and repair the state, then call pthread_mutex_consistent() before unlocking. Calling consistent does not repair any application data itself.

If recovery is impossible, unlocking without marking consistency makes subsequent acquisitions fail with ENOTRECOVERABLE. Such an acquisition does not give the caller ownership. Do not feed robust lock results into the earlier generic must() helper: it treats EOWNERDEAD as an ordinary failure and loses the recovery opportunity. The robust mutex interface defines these transitions.

After owner death, EOWNERDEAD means the new caller owns the mutex: repair the data and call pthread_mutex_consistent before unlocking; unlocking without marking consistency instead makes later locks return ENOTRECOVERABLE without ownership.

  • states: [{"id": "available", "label": "Consistent; unlocked"}, {"id": "owned", "label": "Consistent; mutex owned"}, {"id": "ownerdead", "label": "Owner died; protected state inconsistent"}, {"id": "recovery", "label": "Recovery caller owns mutex; state inconsistent"}, {"id": "unrecoverable", "label": "Not recoverable; no caller ownership"}]
  • transitions: [{"from": "available", "to": "owned", "event": "pthread_mutex_lock returns 0"}, {"from": "owned", "to": "available", "event": "Owner unlocks"}, {"from": "owned", "to": "ownerdead", "event": "Owner dies while holding mutex"}, {"from": "ownerdead", "to": "recovery", "event": "pthread_mutex_lock returns EOWNERDEAD; ownership acquired"}, {"from": "recovery", "to": "owned", "event": "Repair protected data, then pthread_mutex_consistent"}, {"from": "recovery", "to": "unrecoverable", "event": "Unlock without marking consistency"}, {"from": "recovery", "to": "ownerdead", "event": "Recovery owner dies before marking consistency"}, {"from": "unrecoverable", "to": "unrecoverable", "event": "pthread_mutex_lock returns ENOTRECOVERABLE; no ownership"}]
Going deeperRobust ownership is not a transaction

The kernel can report that an owner died. It cannot know whether the owner had updated a queue slot, its index, or its count. Recovery needs a state representation that can be checked, such as explicit operation records or a reconstructible index. Robustness does not guarantee persistent writes reached storage. Sharing a mutex between processes also requires shared memory and PTHREAD_PROCESS_SHARED; robustness alone does not make an ordinary private mutex interprocess-safe.

Cancellation is a different mechanism. pthread_cancel(t) requests cancellation; it does not wait for the target to finish. A new thread normally has cancellation enabled with deferred delivery. The request takes effect at a cancellation point, such as pthread_cond_wait(), read() or pthread_join(). A long computing loop can use pthread_testcancel() at deliberate safe points.

pthread_mutex_lock() is not a cancellation point. A request therefore does not reliably release a worker blocked acquiring a mutex. Asynchronous cancellation permits interruption at almost any point and is unsuitable for ordinary resource-managing code. Prefer a protected stop flag and condition broadcasts for a queue's planned shutdown. See cancellation state and type.

Cancellation during a condition wait reacquires the mutex before invoking cleanup handlers. Without an unlock handler, the canceled thread leaves that mutex owned. This fragment demonstrates the pattern with a separate stop condition and default deferred cancellation:

c
static pthread_mutex_t stop_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t stop_changed = PTHREAD_COND_INITIALIZER;
static int stop_requested;

static void unlock_cleanup(void *argument)
{
    must(pthread_mutex_unlock(argument), "cleanup unlock");
}

static void *cancellable_waiter(void *unused)
{
    (void)unused;
    must(pthread_mutex_lock(&stop_lock), "stop lock");
    pthread_cleanup_push(unlock_cleanup, &stop_lock);
    while (!stop_requested)
        must(pthread_cond_wait(&stop_changed, &stop_lock), "stop wait");
    pthread_cleanup_pop(1);
    return NULL;
}

The coordinator changes stop_requested under stop_lock and broadcasts for normal completion. The handler unlocks on cancellation during the wait. pthread_cleanup_pop(1) also runs it on normal completion. Push and pop can be macros introducing braces, so pair them in one lexical scope; do not return or jump out between them.

Cleanup handlers run in reverse registration order during cancellation or pthread_exit(). A plain return from the start function does not run still-registered handlers. After requesting cancellation, join a joinable target and compare its returned pointer with PTHREAD_CANCELED. Temporarily disabling cancellation around a short multi-resource transition can help, but preserve the old state and establish cleanup before reenabling. Cleanup must restore application invariants as well as unlock objects.

Recovering from a dead mutex ownerOwner diesOwned, inconsistentOwned, consistentUnlocked, usableNot recoverableTransitionsLock returns EOWNERDEADOwner diesOwned, inconsistentRepair; mark consistentOwned, inconsistentOwned, consistentUnlockOwned, consistentUnlocked, usableUnlock without consistencyOwned, inconsistentNot recoverableOwner diesOwned, inconsistentOwned, consistentUnlocked, usableNot recoverableTransitionsLock returns EOWNERDEADOwner diesOwned, inconsistentRepair; mark consistentOwned, inconsistentOwned,consistentUnlockOwned,consistentUnlocked,usableUnlock without consistencyOwned, inconsistentNotrecoverable
Owner death leads to repair and consistency or to an unrecoverable mutex after unlocking without repair.
10

Diagnose mistakes and choose the next primitive

5 min read

Deadlock prevents progress through blocked dependencies. Worker A can hold X while waiting for Y, as worker B holds Y while waiting for X. Relocking a NORMAL mutex can also deadlock one thread.

Document a total lock order, such as queue lock before statistics lock, and follow it everywhere. Include callbacks and error paths. A worker joining another while holding a mutex that the other needs can deadlock without two different locks. Avoid that join-under-lock pattern.

Livelock means activity without useful completion, as workers repeatedly retry incompatible trylocks. Starvation denies one worker a resource while others progress. Ordinary mutexes promise neither FIFO service nor bounded waiting.

Symptom Likely cause Diagnose Fix
Incorrect totals Unprotected shared update Run TSan; inspect both access stacks Protect all accesses or use a suitable atomic
Wrong task arguments Reused loop-index address Inspect argument pointers and lifetimes Retain distinct task objects
Consumer sleeps with work ready Predicate checked outside lock Trace predicate checks and notifications Use one mutex and atomic condition wait
Consumers access an empty queue Single if before waiting Inspect wake-and-reacquire path Loop on the predicate
Shutdown hangs Only one waiter awakened Inspect all waiting groups Close and broadcast both conditions
Worker stops inside a lock Cancellation without cleanup Inspect cancellation points and owners Register cleanup or use cooperative stop
Low throughput Long critical sections or excessive sharing Measure lock hold time and CPU use Move processing outside locks; batch carefully

1Capture all thread states

Interrupt a stuck program in GDB. Use info threads and thread apply all bt to inspect every stack. Connect waiters to required owners or coordinators. Futex waiters may legitimately await future input; waiting alone does not prove deadlock.

On Linux, /proc/PID/task/ lists the process's TIDs; replace PID with the actual numeric process ID. With procps, ps -L -p PID -o pid,tid,stat,comm shows per-thread state. BusyBox ps may lack these options, so use /proc or a fuller diagnostic image.

Tracing adds evidence. strace -f -e trace=futex ./queue_demo follows thread activity and shows futex waits and wakes. Install strace where available. No visible futex call does not prove the program used no mutexes: fast-path operations stay in user space. Tracing also changes timing.

2Detect a deliberate race

For the race lab, adapt the earlier example: two workers increment one shared ordinary counter without locks. Read it after joining both. Keep lifetimes and creation handling correct.

Build with ThreadSanitizer, or TSan, which instruments accesses and tracks synchronization:

bash
gcc -std=c11 -Wall -Wextra -g -O0 -pthread -fsanitize=thread \
    race.c -o race_tsan
./race_tsan

A report identifies conflicting accesses and thread stacks. Fix synchronization and rerun. GCC does not combine thread and address sanitizers; use separate builds. A clean run cannot prove freedom from deadlock, lost wakeups or unexecuted races.

TSan needs substantial memory and a supported runtime. Unexpected-memory-mapping startup failures mean it did not analyze your program. Some WSL2/runtime combinations have failed; WSL2 is not universally incompatible. Retry on compatible Linux and retain environment details. Consult the TSan documentation for runtime limits.

3Choose alternatives by their rules

A read-write lock permits simultaneous readers or one exclusive writer. Use pthread_rwlock_init(), pthread_rwlock_rdlock(), pthread_rwlock_wrlock(), pthread_rwlock_unlock() and eventual destruction. Readers must truly avoid modifying protected state. Counting hits inside a read section is still a write.

Do not request a write lock while holding its read lock. Release, acquire for writing and recheck state. Writer admission and fairness depend on implementation and scheduling. Bookkeeping and cache contention can outweigh parallel reading; consult read-lock rules.

A barrier is a rendezvous: a configured number of participants must arrive before the phase can continue. Initialize using pthread_barrier_init(&b, NULL, participants) and call pthread_barrier_wait(&b) each phase. One unspecified participant receives PTHREAD_BARRIER_SERIAL_THREAD; the others receive zero. Both mean success, so the earlier must() helper is unsuitable without handling that special return.

If that chosen participant computes a shared result after the wait, peers need another synchronization step before reading it. A second barrier can mark completion. A missing participant leaves the others waiting; handle startup failure before entering the phase protocol. Destroy only after all users finish. See barrier semantics.

Measure this queue before replacing its mutex. The next lesson uses single-producer, single-consumer rings: each index has one writer, and publication orders payload access. More producers require a different algorithm.

Find the missing progress stepCapturestacksInspecteverythreadFindrequiredstateLock, item,space orcompletionIdentifyownerWho canchange thatstate?Follow dependenciesFindmissingaction orcycleRepairprotocolRecheckwith stressandanalysisCapturestacksInspecteverythreadFindrequiredstateLock, item,space orcompletionIdentifyownerWho canchange thatstate?Follow dependenciesFindmissingaction orcycleRepairprotocolRecheckwith stressandanalysis
Debugging connects each blocked thread to the state change and owner it needs.
11

Try it

1 min read

Open this lesson's lab and practice pack to build a queue with many producers and consumers. Stress empty, full and closing states; show that every accepted item is consumed exactly once. Extend it into a reusable thread pool with documented ownership and coordinator-owned shutdown. Reproduce a race using TSan and explain your fix before continuing to Advanced synchronisation, atomics, lock-free ring buffer.

Key takeaways

  • A separate stack gives each thread working space, not memory isolation.
  • Keep argument and result objects alive until their last user has finished.
  • Use one documented mutex for every access to each protected invariant.
  • A condition variable carries wakeups; the protected predicate carries meaning.
  • Recheck predicates in while loops after every condition wait.
  • Close a queue, wake affected waiters, join workers, then destroy shared objects.
  • Uncontended ordinary mutexes usually stay in user space; futexes support blocking.
  • Cancellation, owner death and scheduling priority need explicit application policies.

Reflect

  1. Explain to a colleague, in two sentences: distinguish process resources, per-thread state, Linux TIDs and pthread handles.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close