Allocation begins with ownership
4 min readImagine a sensor service receiving messages of different lengths. A fixed array wastes space or rejects larger messages. Dynamic allocation lets the service request space while it runs.
An a component that divides available memory into blocks and tracks which blocks callers may use manages those requests. The preceding lesson explained how Linux provides virtual memory through mappings and the program break. Here you divide that memory into smaller objects.
the responsibility to keep an allocation alive while needed and release it exactly once afterward belongs to your program. A pointer does not carry an owner label. Copying a pointer creates another way to access the same object, not another allocation.
These library calls are declared in <stdlib.h>:
| Call | What you may rely on |
|---|---|
malloc(n) |
Returns space for at least n bytes, or NULL on failure; bytes are uninitialized. |
calloc(count, size) |
Allocates an array and sets all its bytes to zero; detects multiplication overflow. |
free(p) |
Releases a valid live allocation; free(NULL) does nothing. |
realloc(p, n) |
Resizes an allocation, possibly moving it, while preserving the smaller of the old and new sizes. |
Zero bytes represent integer zero, but C does not promise that they represent every pointer or floating-point zero. Initialize those types explicitly when portability matters. On Linux, allocation failure normally sets errno to ENOMEM; ISO C alone does not require that error-reporting rule.
Use a temporary pointer when resizing. This fragment assumes a live byte buffer buf, its recorded length len, and a positive new_len:
unsigned char *candidate = realloc(buf, new_len);
if (candidate == NULL) {
/* buf and its first len bytes are still valid. */
report_allocation_failure();
} else {
buf = candidate;
len = new_len;
}
The reporting function is application code, not a library interface. Additional bytes after growth are uninitialized. After success, rebuild all pointers into the object using saved offsets. The old pointer is invalidated even when the returned address has the same numerical value.
Never pass an interior pointer, stack address or already freed pointer to free. These errors have undefined behaviour: C imposes no useful result. A dangling pointer refers to an object whose lifetime has ended. Setting one pointer to NULL after freeing it helps that variable, but leaves other aliases dangling.
Do not use realloc(p, 0) as portable shorthand for free(p). Its treatment changed across C standards and libraries; C23 makes it undefined behaviour. Handle zero sizes yourself. For malloc(0) and zero-size calloc, a permitted non-null result still gives you no bytes to access.
Alignment and the two kinds of waste
3 min readSuppose you request 17 bytes and the allocator rounds this to 32. The extra 15 bytes are part of the allocator's space cost. They do not extend your requested object's bounds.
the requirement that an object's address be a multiple of a specified number of bytes lets the processor access objects correctly. Standard allocation supports ordinary types with fundamental alignment. It does not promise every larger alignment requested for specialized objects.
glibc is the GNU C library, Ubuntu's implementation of standard C interfaces. Its allocator on common x86-64 and AArch64 Linux systems provides 16-byte alignment. Do not turn this into a universal C rule. A 32-bit ARM library may have different requirements. Check the target's application binary interface, the rules used by compiled code to exchange data and call functions.
Our teaching allocator deliberately provides 16-byte alignment. It checks at compilation that this covers max_align_t, C's type for fundamental alignment. The payload is the space your code receives. A allocator-owned metadata recording a block's capacity, state and links sits before each payload in our design. Rounding the header size to 16 preserves payload alignment; rounding only the payload cannot fix a misaligned header.
For a power-of-two alignment A, rounding uses (n + A - 1) & ~(A - 1). Check the addition first. Unsigned overflow wraps and can turn a huge request into a tiny block. That tiny allocation may succeed before the caller writes the original huge amount.
space consumed inside allocated blocks by rounding and other unused capacity increases when size classes are coarse. Headers add another space cost, which some accounting schemes report separately. A program holding many tiny objects can spend more on bookkeeping than useful data.
free space divided into separate holes that cannot satisfy a larger contiguous request depends on allocation lifetimes. Freeing alternate blocks leaves holes separated by live objects. Adding the holes' sizes does not make one larger hole.
Record useful bytes, reserved block capacity, total free capacity and the largest free block separately. A small largest-free-block value explains failure even when total free space looks healthy. Resident memory, the pages presently in RAM, measures something different again.
For larger alignment, posix_memalign(&p, alignment, size) requires a power of two that is a multiple of sizeof(void *). It returns an error number directly. C11 aligned_alloc(alignment, size) also requires size to be a multiple of alignment. Successful results from both are released with free.
An aligned user-space pointer does not guarantee physically contiguous memory, a device-visible address or consistent CPU and device views of data. Direct memory access, where a device transfers data without CPU copying, needs the driver's supported buffer and mapping interface.
Obtain backing memory and describe each block
3 min readA an allocator that advances a cursor after each allocation and normally reclaims everything together is the simplest starting point. Align the cursor, check remaining capacity, return it and advance. Individual free calls cannot recover arbitrary holes without more bookkeeping.
We will instead track every block in address order. Each header sits before its payload. cap records rounded capacity; used records the requested length. Keeping both makes resizing and waste measurement clearer.
1Map a bounded region
Create mini_alloc.c and begin with this listing. The later core sections append to the same file. The chosen region is 64 KiB for the exercise, not a claim about Linux page size.
#define _DEFAULT_SOURCE
#include <assert.h>
#include <errno.h>
#include <stddef.h>
#include <stdint.h>
#include <string.h>
#include <sys/mman.h>
#define ALIGNMENT ((size_t)16)
#define POOL_BYTES ((size_t)64 * 1024)
typedef struct block Block;
struct block {
size_t cap;
size_t used;
int is_free;
Block *next;
};
#define HEADER_BYTES \
((sizeof(Block) + ALIGNMENT - 1) & ~(ALIGNMENT - 1))
_Static_assert(_Alignof(max_align_t) <= ALIGNMENT,
"target needs stronger fundamental alignment");
_Static_assert(ALIGNMENT % _Alignof(Block) == 0,
"header alignment is unsupported");
static Block *head;
static int rounded_size(size_t n, size_t *out)
{
if (n > (size_t)PTRDIFF_MAX || n > SIZE_MAX - (ALIGNMENT - 1)) {
errno = ENOMEM;
return 0;
}
*out = (n + ALIGNMENT - 1) & ~(ALIGNMENT - 1);
return 1;
}
static unsigned char *payload(Block *b)
{
return (unsigned char *)b + HEADER_BYTES;
}
static int initialize_pool(void)
{
if (head != NULL)
return 1;
void *area = mmap(NULL, POOL_BYTES, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (area == MAP_FAILED)
return 0;
head = area;
*head = (Block){ POOL_BYTES - HEADER_BYTES, 0, 1, NULL };
return 1;
}
Linux anonymous mappings begin on page boundaries, satisfying our 16-byte assumption on the intended targets. _DEFAULT_SOURCE exposes the mapping flag with glibc in strict C compilation. The code assumes Linux's ordinary flat address space, not a freestanding microcontroller environment.
The mapping initially contains one free block. mmap returns MAP_FAILED on failure, not NULL. Zero-filled backing memory does not mean future allocations are zero-filled: freed blocks will contain old data.
2Keep backing allocation separate from suballocation
brk changes the program break; Linux's sbrk library wrapper adjusts it by an increment. sbrk reports failure as (void *)-1. These are legacy interfaces, absent from modern POSIX, although Linux still provides them.
Do not mix your own break management with glibc's allocator. Libraries can allocate without your knowledge. Independent mmap regions keep this exercise separate from that shared process-wide resource.
One mapping per tiny allocation would avoid some bookkeeping but add mapping and page-rounding costs. Our single mapping amortizes that cost. The tradeoff is explicit: exhaustion returns ENOMEM; this version never grows the region.
Search, split and coalesce
4 min readA a linked structure used to find reusable blocks can contain only free blocks or include all blocks with state flags. Our address-ordered list uses the second design. It is simple enough to inspect in a debugger.
First-fit picks the first suitable free block. Best-fit searches for the smallest suitable one. A full linked-list search takes time proportional to the blocks inspected. Best-fit can leave tiny leftovers; it does not universally minimize fragmentation.
1Divide a block only when both pieces are useful
Append these helpers. split_block receives an aligned size no larger than b->cap. A remainder needs its own header and at least one alignment unit of payload.
static void split_block(Block *b, size_t want)
{
if (b->cap - want < HEADER_BYTES + ALIGNMENT)
return;
Block *tail = (Block *)(payload(b) + want);
*tail = (Block){ b->cap - want - HEADER_BYTES, 0, 1, b->next };
b->cap = want;
b->next = tail;
}
static void merge_right(Block *b)
{
while (b->next != NULL && b->next->is_free &&
payload(b) + b->cap == (unsigned char *)b->next) {
Block *right = b->next;
b->cap += HEADER_BYTES + right->cap;
b->next = right->next;
}
}
void *mini_malloc(size_t n)
{
size_t want;
if (n == 0)
return NULL;
if (!rounded_size(n, &want) || !initialize_pool())
return NULL;
for (Block *b = head; b != NULL; b = b->next) {
if (b->is_free && b->cap >= want) {
split_block(b, want);
b->used = n;
b->is_free = 0;
return payload(b);
}
}
errno = ENOMEM;
return NULL;
}
void mini_free(void *p)
{
if (p == NULL)
return;
Block *b = (Block *)((unsigned char *)p - HEADER_BYTES);
assert(!b->is_free);
b->is_free = 1;
b->used = 0;
for (Block *cur = head; cur != NULL; cur = cur->next)
if (cur->is_free)
merge_right(cur);
}
2Merge physical neighbours
combining physically adjacent free blocks into one larger free block recovers their separating header as usable capacity. mini_free scans from the beginning, so it can merge both preceding and following free blocks. merge_right keeps merging until the next block is occupied or absent.
The address equality is essential if you later add multiple mappings. Being next in a free-only list does not establish physical adjacency. Never merge across a live block or a mapping boundary.
Round: Validate the requested size, then round capacity to the alignment.
Search: Walk the block list until a free block has sufficient capacity.
Split: Leave a second free block only if its header and payload fit.
Own: Record the requested size and mark the returned block occupied.
Release: Mark the original block free when its owner finishes.
Join: Merge physically adjacent free neighbours for later larger requests.
Our list has an important invariant, a condition that must always remain true: it partitions the entire region without gaps or overlaps. Every capacity is aligned, every header fits, and two free neighbours cannot remain after cleanup. Check these properties during torture testing.
A thread is one execution path within a process. Use this allocator only from one thread, with valid pointers from its own API. Do not rename it malloc, preload it into applications or call it from signal handlers. Its assertion catches some repeated frees, not arbitrary invalid pointers. It is neither a secure allocator nor a full libc replacement.
Resize safely and diagnose common mistakes
4 min readrealloc is a separate challenge because it changes both size and potentially location. Shrinking can split off a tail. Growing can use an adjacent free block or allocate elsewhere. Failure must preserve the original object.
1Handle each resize case explicitly
Append this implementation. It deliberately uses allocate-copy-free for growth beyond current capacity. This is slower than growing into a neighbour, but keeps failure behaviour easy to verify.
void *mini_realloc(void *p, size_t n)
{
if (p == NULL)
return mini_malloc(n);
if (n == 0) {
mini_free(p);
return NULL;
}
size_t want;
if (!rounded_size(n, &want))
return NULL;
Block *b = (Block *)((unsigned char *)p - HEADER_BYTES);
assert(!b->is_free);
if (want <= b->cap) {
split_block(b, want);
if (b->next != NULL && b->next->is_free)
merge_right(b->next);
b->used = n;
return p;
}
void *replacement = mini_malloc(n);
if (replacement == NULL)
return NULL;
memcpy(replacement, p, b->used);
mini_free(p);
return replacement;
}
void *mini_calloc(size_t count, size_t size)
{
if (size != 0 && count > SIZE_MAX / size) {
errno = ENOMEM;
return NULL;
}
size_t bytes = count * size;
void *p = mini_malloc(bytes);
if (p != NULL)
memset(p, 0, bytes);
return p;
}
int mini_destroy(void)
{
for (Block *b = head; b != NULL; b = b->next) {
if (!b->is_free) {
errno = EBUSY;
return -1;
}
}
if (head != NULL && munmap(head, POOL_BYTES) == -1)
return -1;
head = NULL;
return 0;
}
The custom API defines mini_realloc(p, 0) as release-and-return-null. This is our rule, not a claim about standard realloc. It also chooses NULL for zero-byte allocation. The destroy function releases the mapping only after every object has been freed.
Growth copies b->used bytes because this branch guarantees the new size exceeds old capacity. Shrinking preserves the first n bytes in place. Never copy the requested new size from an old smaller object.
2Turn symptoms into checks
| Symptom | Likely cause | Diagnose | Fix |
|---|---|---|---|
| Huge request gets a tiny block | Size arithmetic wrapped | Test requests near SIZE_MAX |
Check before adding or multiplying |
| Alignment fault on ARM | Header displaced payload | Check returned addresses modulo alignment | Round header and payload sizes |
| Large allocation fails after many frees | Free holes remain separate | Walk blocks and inspect largest hole | Coalesce adjacent free blocks |
| Data disappears on failed resize | Owner pointer overwritten | Force pool exhaustion | Assign through a temporary pointer |
| Crash occurs during a later free | Earlier heap overwrite damaged metadata | Run instrumented reproduction | Correct the first invalid write |
| Memory stays high after free | Allocator retains reusable storage | Compare live bytes with mapped memory | Measure before changing retention policy |
For neighbour-based growth, first check whether combined capacity is sufficient without changing the list. Merge only once success is assured, then split any tail. Otherwise a failed resize might quietly alter allocator state in surprising ways.
Compile the assembled implementation as an object before linking your own driver. This checks declarations and warnings without requiring a main function:
gcc -std=c11 -Wall -Wextra -Wpedantic -g -O0 -c mini_alloc.c
A successful command writes mini_alloc.o. It does not prove correctness. Your driver should maintain many live allocations, fill each with a distinct byte pattern, resize them, then verify every preserved prefix. Use a repeatable random seed and print the operation index on failure.
Include exact fits, unsplittable tails, exhaustion, overflow requests, all free orders and reuse after complete cleanup. Force resize failure while checking the old data. Do not dereference deliberately invalid pointers in the normal correctness suite; isolate undefined-behaviour demonstrations in separate processes.
Memcheck follows addresses and defined values
4 min readA memory leak is an allocation you no longer need but never release. A use-after-free accesses an object after release. A double free releases it twice. A heap overflow crosses an allocated object's boundary.
A stack buffer overflow crosses a local array's boundary. Stack exhaustion instead consumes the thread's available stack, often through deep recursion. An uninitialized read uses bytes before your program has assigned meaningful values. These failures need different evidence even when all eventually crash.
Memcheck runs machine code through Valgrind's instrumentation engine, which adds checks while executing translated instructions. It tracks addressability and whether individual value bits are defined. A copied uninitialized value may be reported only when it affects a branch, address or system call.
1Build a separate native debug executable
Save this deliberately incorrect program as heap_bug.c in your lab workspace. Argument u selects a use-after-free; l selects a leak. The branches keep the two reports independent.
#include <stdlib.h>
static void lose_buffer(void)
{
unsigned char *p = malloc(37);
if (p == NULL)
exit(2);
p[0] = 7;
}
int main(int argc, char **argv)
{
if (argc != 2)
return 2;
if (argv[1][0] == 'l') {
lose_buffer();
return 0;
}
if (argv[1][0] == 'u') {
unsigned char *p = malloc(37);
if (p == NULL)
return 2;
p[0] = 7;
free(p);
return p[0]; /* Deliberate lifetime error. */
}
return 2;
}
Use -O0 for this small demonstration. Optimization may remove buggy operations whose results do not matter. The warning about the freed pointer is useful; warnings remain part of the debugging process.
gcc -g -O0 -Wall -Wextra -o heap_bug heap_bug.c
valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all \
--track-origins=yes --error-exitcode=99 ./heap_bug u
valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all \
--track-origins=yes --error-exitcode=99 ./heap_bug l
For the invalid read, inspect the access stack, freeing stack and allocation stack. Fix the earliest cause, then rerun. Later errors may be consequences. --track-origins=yes helps trace undefined values back to their source; it costs additional time and memory.
2Interpret leak reachability
| Category | Meaning | First investigation |
|---|---|---|
| Definitely lost | No pointer chain to the block remains | Find where the last owning pointer disappeared |
| Indirectly lost | Reachable only through another lost block | Repair the lost parent first |
| Possibly lost | A chain includes an interior pointer | Check intentional interior pointers and pointer corruption |
| Still reachable | A pointer chain still reaches the block | Decide whether lifetime and retention are intentional |
Reachability is a tool's inference from memory, not proof of useful ownership. A global cache can remain reachable while growing without bound. Conversely, a short-lived command can intentionally keep a bounded table until exit.
--error-exitcode=99 makes detected errors visible to automation. Leak kinds counted as errors are separately controlled by --errors-for-leak-kinds; use definite,indirect for a focused gate or all for stricter review. Record suppressions, rules that hide known reports, and keep them narrowly scoped.
Going deeperTeach Memcheck about your allocator
Our mapping is accessible memory, but Memcheck does not automatically know each mini_malloc object's bounds or lifetime. Valgrind client requests such as VALGRIND_MALLOCLIKE_BLOCK and VALGRIND_FREELIKE_BLOCK describe custom objects. Memory-pool requests support region allocators. Mark allocation state correctly, including newly exposed resize bytes; do not tell the tool uninitialized bytes are initialized. Keep allocator headers addressable to allocator code while protecting client payload lifetimes. A clean unannotated run validates much less than a clean standard-malloc run.
ASan checks poisoned boundaries
3 min readAddressSanitizer, usually called ASan, combines compiler instrumentation with a runtime library. A a deliberately inaccessible padding region around an object that helps detect boundary crossings catches many overflows. Poisoning marks storage forbidden in the checking metadata. Released allocations are poisoned and often delayed in quarantine before reuse, improving use-after-free detection.
1Compile and link with instrumentation
Build another executable from the same source. The sanitizer flags must also appear at the final link step, so the compiler driver adds the runtime.
gcc -g -O0 -fno-omit-frame-pointer -fsanitize=address,undefined \
-o heap_bug_asan heap_bug.c
ASAN_OPTIONS=detect_leaks=1 ./heap_bug_asan u
ASAN_OPTIONS=detect_leaks=1 ./heap_bug_asan l
Read the error kind, access location and allocation/free stacks. ASan normally stops at the first detected error; the separate leak run must reach normal termination. LeakSanitizer, where supported, examines allocations at exit and is integrated into ASan on supported Linux targets.
-fsanitize=undefined adds UndefinedBehaviorSanitizer, or UBSan. It catches selected language violations such as signed integer overflow and misaligned accesses. It does not turn every undefined operation into a diagnosed error. Ordinary unsigned overflow is defined arithmetic, so retain explicit allocation-size checks.
2Follow shadow memory
a separate address region storing compact metadata about the accessibility of application memory makes per-access checking practical. In a common ASan mapping, one shadow byte describes eight application bytes. The mapping has the form shadow = (address >> 3) + offset.
The scale is three for that mapping. The offset belongs to the target runtime and address-space layout; do not copy a desktop constant into an ARM explanation. ASan supports target-specific layouts, and virtual address reservation is not the same as resident RAM usage.
Zero shadow means the whole described group is accessible. Small positive values describe a partially accessible final group. Special poison values mark redzones, freed regions and other forbidden areas. Compiler-generated checks account for access size and position within the group.
Allocate: The runtime reserves the object and surrounding redzones.
Describe: It marks payload bytes accessible and boundary bytes poisoned in shadow memory.
Translate: An instrumented access computes the corresponding shadow location.
Check: Generated code tests the metadata for the access range.
Report: A forbidden access produces a diagnostic and allocation history.
Quarantine: Freeing poisons the payload and may delay reuse of its storage.
ASan detects many heap and stack boundary errors, double frees and invalid frees. It does not generally detect uninitialized-value use. It can also miss an overflow that stays inside a larger addressable object or lands in another valid object. Uninstrumented code adds further blind spots.
Memcheck may miss a stack array overflow that stays inside addressable stack space. ASan's instrumented stack redzones often expose it. For stack exhaustion, inspect recursion in a debugger backtrace and review automatic array sizes. Increasing the stack limit can delay failure without fixing unbounded recursion.
Going deeperCustom allocation changes what a clean ASan run means
ASan sees our mmap region, not automatically its smaller objects. Compiler checks alone do not poison every custom boundary or mark mini_free payloads inaccessible. Integration uses <sanitizer/asan_interface.h>, including __asan_poison_memory_region and __asan_unpoison_memory_region. Shadow granularity complicates partial boundaries. Headers must remain accessible to allocator code, and split, merge, free and resize paths must update poisoning consistently. Test these annotations with intentional failing cases before trusting the allocator's sanitizer results.
Run Memcheck on the ordinary build and ASan on its instrumented build. Running the ASan executable under Memcheck combines competing memory mechanisms and obscures the comparison.
glibc trades simple lists for fast common paths
4 min readOur allocator walks every block and has no concurrency protection. glibc uses several structures instead. An in glibc, an allocator state with its own bookkeeping and synchronization for serving allocations helps multiple threads allocate without one global lock on every operation.
The main arena can obtain backing through the program break. Additional arenas use mapped memory. Large allocations can receive separate mappings, but size alone does not predict every call's backing. Existing reusable blocks and tuning settings also matter.
glibc groups free chunks into bins, collections organized by size and reuse rules. Chunk headers encode sizes and status bits. Some metadata describes a preceding free chunk, allowing efficient backward merging. This is richer than our explicit size-plus-free-flag design.
In glibc source, begin with malloc/malloc.c. Follow __libc_malloc through cache and arena selection, then _int_malloc for arena allocation logic. Read the source matching your deployed library; internal helper names and paths can evolve. These helpers are not public interfaces for application code.
The thread cache, called tcache, keeps eligible freed chunks near the thread that released them. Common allocation/free pairs can avoid arena locking. Cached chunks are not necessarily coalesced immediately. A pointer freed by another thread may enter that thread's cache, so allocation history and current cache ownership can differ.
An arena is not necessarily dedicated permanently to exactly one thread. More arenas can reduce lock contention yet increase retained memory. More caching can improve speed while hiding reusable bytes from other allocation paths.
Inspect glibc-specific statistics with malloc_info, which emits allocator information as XML, or mallinfo2, whose counters avoid the narrow integer fields of old mallinfo. Neither reports your application's logical ownership perfectly. Interpret them alongside live-object counts and /proc/PID/smaps_rollup, which summarizes mapped memory accounting.
1Separate debugging controls from normal hardening
MALLOC_CHECK_ is a glibc debugging control found in older guides. On glibc builds using the separately provided malloc debug library, preload libc_malloc_debug.so.0 for this facility; the environment variable alone is insufficient. Library availability and loader paths depend on the distribution.
The related glibc.malloc.check tunable also depends on that debug support. These mechanisms catch selected allocator mistakes, not all invalid accesses. Ordinary glibc already performs some consistency checks, including checks that can abort on a double free. Their presence does not require MALLOC_CHECK_ to be enabled.
mtrace() from <mcheck.h> enables allocation tracing to the file named by MALLOC_TRACE. muntrace() stops it; the mtrace analysis program processes the log. glibc releases that moved malloc debugging out of the main library require the debug library for tracing too. Do not assume the 2010 book's setup still works unchanged.
2Tune one policy while measuring
mallopt from <malloc.h> changes glibc policies. It returns nonzero for success and zero for failure. As a controlled startup experiment, before other allocations, this fragment sets a mapping threshold:
if (mallopt(M_MMAP_THRESHOLD, 256 * 1024) == 0)
return EXIT_FAILURE;
The selected value is an experiment, not a recommendation. Requests at or above the threshold may use individual mappings when existing free space cannot satisfy them. Such mappings are easier to return independently. They also add mapping work and page-rounding overhead.
glibc can adjust its threshold dynamically; explicitly setting it disables that automatic adjustment. M_ARENA_MAX and glibc.malloc.arena_max bound arena creation. glibc.malloc.tcache_count controls per-bin cached counts. These settings belong to glibc; other C libraries, such as musl, use different allocator internals and controls.
Profile peaks, churn and contention
3 min readBegin with a real workload: for example, a telemetry service receiving bursts, draining its queue and returning to idle. Peak live bytes, allocation count and retained idle memory describe different failure risks. Keep input data and configuration fixed when comparing builds.
1Read the peak with Massif
Massif samples heap use and records allocation call trees. Run your native debug build, replacing the executable and workload arguments with your real program:
valgrind --tool=massif --time-unit=B --massif-out-file=massif.out \
./telemetry_service --replay sample.bin
ms_print massif.out
B uses allocated/deallocated byte activity as the time axis, not wall-clock seconds. ms_print shows snapshots and a heap-use graph. Find the peak snapshot, then follow its allocation tree to the largest retained callers. A peak in message buffering may be caused by slow consumers, not an inefficient allocator.
Massif separates useful heap bytes from estimated administrative overhead and optional stack measurements. It does not equal process resident memory. --stacks=yes includes stack profiling. --pages-as-heap=yes switches to page-oriented profiling for mappings; use a separate run because the measured object changes.
2Distinguish retention from churn
DHAT, Valgrind's dynamic heap analysis tool, measures allocation behaviour including block lifetimes and accesses. Many short-lived allocations can produce modest peaks but heavy allocation traffic. DHAT can expose objects that are allocated repeatedly or barely accessed.
Cachegrind counts executed instructions and can simulate cache and branch behaviour. Callgrind adds call relationships for attribution. Their simulated costs are not elapsed time or direct measurements of the board's cache hierarchy. Validate conclusions on the actual target when cache size, instruction set or memory latency matters.
These separate runs collect different evidence from the same native program:
valgrind --tool=dhat ./telemetry_service --replay sample.bin
valgrind --tool=cachegrind --cache-sim=yes --branch-sim=yes \
./telemetry_service --replay sample.bin
valgrind --tool=callgrind ./telemetry_service --replay sample.bin
valgrind --tool=helgrind ./telemetry_service --replay sample.bin
Use DHAT's viewer for its output, cg_annotate for Cachegrind and callgrind_annotate for Callgrind. Helgrind checks synchronization and potential data races, where concurrent accesses conflict without appropriate ordering. It does not replace heap-boundary checking. We develop the thread model in the next lesson.
ThreadSanitizer uses compiler instrumentation through -fsanitize=thread to detect data races on supported targets. Build it separately: it cannot be combined with AddressSanitizer or standalone LeakSanitizer. Its runtime and address-space requirements can exclude small or unsupported targets. A clean run covers only executed schedules and understood synchronization.
Standalone -fsanitize=leak links LeakSanitizer without ASan's access checking on supported configurations. It does not detect buffer overflows or use-after-free accesses. Leak analysis can be conservative, and debugger or tracing arrangements can interfere with its operation.
Use a host reproduction for rapid bug finding when the board lacks RAM or runtime packages. Then repeat relevant tests on the target. Cross-compilation needs sanitizer libraries for the target architecture and C library; copying a host runtime does not work. BusyBox supplies utilities, not Valgrind or sanitizer runtimes.
Choose bounded designs and understand hardening
4 min readFor one decoded message, allocate its temporary objects from a region and reset the region after processing. Here “arena” means a lifetime region, not glibc's lock-protected allocator state. Bump allocation makes the common path short, but a pointer escaping the reset becomes invalid.
A slab or fixed-size pool divides storage into equal-size slots. A free-slot list can allocate and release in constant bookkeeping time. It avoids external fragmentation within that class, but wastes space when object sizes differ and can strand capacity in the wrong pool.
Round each slot stride to satisfy both object size and alignment. Align the backing base too. Metadata stored in a free slot must itself fit and remain aligned. For a required alignment A, calculate an aligned start after the header and retain enough information to recover the original allocation on release.
Per-thread regions or pools reduce shared-state traffic. They complicate cross-thread frees: decide whether to queue returns to the owner, synchronize a shared pool or transfer ownership. Padding frequently modified thread state onto separate cache lines can reduce false sharing, where unrelated writes invalidate the same cache line.
Non-uniform memory access, or NUMA, means access cost depends on the memory node serving a CPU. Linux placement policies and the node used on first touch influence locality. A thread allocating a buffer is not enough if a different thread first writes its pages. Use supported NUMA policy interfaces, such as mbind, only on systems with relevant topology, and measure.
For an embedded control path, preallocate bounded storage before entering the critical phase. Define what happens when every slot is occupied. Constant-time list bookkeeping alone does not guarantee bounded latency: page faults, locks, cache misses and scheduling still matter.
Going deeperTcache poisoning, hardening and sampled detection
Tcache poisoning corrupts free-list linkage so a later allocation follows attacker-influenced metadata. It commonly begins with an underlying use-after-free or overwrite. glibc's safe-linking encodes singly linked free-list pointers using address-derived information and checks alignment; additional consistency checks detect some repeated frees. These checks raise barriers but do not make dangling pointers safe. Do not treat pointer encoding as encryption or a complete security boundary.
glibc.malloc.perturb fills allocation and release paths with diagnostic patterns, but caching paths and release details affect coverage. Reducing glibc.malloc.tcache_count, including setting it to zero, can help compare behaviour with less caching; it changes performance and reuse patterns rather than repairing bugs. Verify tunable support against the deployed glibc build.
GWP-ASan samples a subset of allocations into guarded storage and captures allocation/free history. Guard pages can fault when a sampled access crosses a protected boundary. Sampling keeps overhead suitable for some deployed products, but unsampled objects and some accesses escape detection. It is an integrated allocator/runtime facility, not a universal GCC flag.
GrapheneOS's hardened_malloc is a separate allocator with design measures such as isolated metadata, guarded regions and delayed reuse. Its supported targets, memory overhead and integration requirements matter. Evaluate compatibility and measured behaviour before selecting it for an embedded image. Neither allocator hardening nor sampled diagnostics replaces fixing the original ownership error.
A professional decision starts with evidence: object sizes, simultaneous demand, lifetime groups, cross-thread transfers and failure policy. Compare the simplest suitable design against the general allocator using those same workloads. Keep correctness checks active while changing performance settings.
Try it
1 min readOpen the lab, “Build and torture-test a coalescing allocator; compare Memcheck and ASan; inspect a Massif peak.” Use the practice pack to exercise allocation order, resize failure and preserved data without changing the expected API. Compare the deliberate leak and use-after-free in separate Memcheck and ASan runs, then explain what each report can see. Finish by identifying a real program's peak allocation path in Massif and proposing one measurable improvement.
Key takeaways
- Free releases ownership; it does not promise to reduce the process memory footprint.
- Align both headers and payloads, and check arithmetic before rounding or multiplying.
- Split only when the remainder can hold a header and a useful aligned payload.
- Coalesce physically adjacent free blocks, never merely neighbouring free-list entries.
- A failed nonzero realloc leaves the original allocation and its contents intact.
- Memcheck tracks definedness; ASan checks instrumented accesses against poisoned shadow memory.
- Custom suballocations need tool annotations before tools can see their individual boundaries.
- Measure allocation lifetimes and peak demand before changing allocator policy.
Reflect
- Explain to a colleague, in two sentences: explain allocation ownership, alignment, zero initialization and realloc failure semantics.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Man pagemalloc(3): allocation interfaces and errors
- Official docsValgrind User Manual: Memcheck and profiling tools
- Official docsClang AddressSanitizer
- Official docsClang UndefinedBehaviorSanitizer
- Official docsClang ThreadSanitizer
- Official docsGNU C Library: Memory Allocation Tunables
- Official docsLLVM GWP-ASan
- Official docsGrapheneOS hardened_malloc
Official documentation first. Links checked when the lesson was published.