Separate data delivery from waking a worker
2 min readImagine a sensor process sending samples to a storage process. Giving both the same numeric pointer does not connect their memory. Each normally has its own virtual address space, the set of addresses it can access.
Interprocess communication, or IPC, lets separate processes exchange data or coordinate their actions. Earlier lessons used pipes between processes and shared objects between threads. Here you connect independent processes without assuming their ordinary variables are shared.
Choose what crosses the boundary: a temperature message, a shared camera frame, or a wakeup announcing completed work.
A message boundary marks where one sent record ends and another begins. Message queues preserve these boundaries. A pipe carries a byte stream, so you define record boundaries. A socket is a communication endpoint; its type determines whether it carries a stream or separate messages.
Shared memory supplies neither record boundaries nor automatic exclusion between writers. You design its layout and ownership rules. The previous lesson proved an atomic single-producer, single-consumer ring. Here you add safe creation and blocking wakeups across processes.
Backpressure makes a producer slow down or reject work when downstream capacity is exhausted. A full queue can block its sender. A full shared ring needs an equivalent rule. Otherwise your design merely moves overload into overwritten samples or unbounded memory growth.
If the consumer disappears, decide whether producers block, discard samples, or report a fault. Telemetry may tolerate loss; commands may require acknowledgment. Enqueueing proves acceptance into storage, not execution.
Small copied messages can outperform shared mappings with complicated coordination. Shared memory avoids a kernel-mediated payload copy on each transfer, but still pays for synchronization and cache traffic. Measure on your embedded target.
An eventfd is a Linux kernel counter accessed through a file descriptor for waiting and notification.
Find objects and understand their lifetime
2 min readLinux supports older System V interfaces and later POSIX interfaces. POSIX is a standard for operating-system interfaces. Both families remain useful.
POSIX queues, shared memory and named semaphores use names such as /learner_samples. Use one leading slash and no further slash for portable names. These names are API names, not arbitrary absolute file paths.
System V creation calls use an integer key of type key_t. They return an integer object identifier. A key finds an object; an identifier refers to that particular object. Permissions control access.
ftok(path, project) derives a key from an existing filesystem object and the low eight bits of a nonzero project value. Check for (key_t)-1. It does not reserve a unique key, and collisions are possible. Replacing the path's file can change its key. The file is a lookup aid, not the queue's backing storage.
IPC_PRIVATE requests a new System V object, without restricting access automatically. Set suitable permissions and distribute its identifier. Creation before fork() gives both processes the identifier.
Kernel persistence means an IPC object can outlive its creating process, without promising survival across a reboot. System V objects normally remain until explicitly removed or their IPC namespace is destroyed. An IPC namespace is an isolated collection of IPC objects and related limits. POSIX queues also have kernel persistence within their namespace.
Linux normally exposes POSIX shared memory under /dev/shm, on tmpfs, a filesystem backed by memory and potentially swap. Named semaphores usually appear there with a sem. prefix. POSIX queues appear under /dev/mqueue when the mqueue filesystem is mounted. Visible filesystem entries do not make these objects persistent disk records.
mq_close() releases your queue handle. mq_unlink() removes its name; existing handles keep working until closed. Shared-memory unlink similarly removes discovery by name, while existing mappings remain usable. Unlinking and recreating a name can therefore leave old and new clients using different objects.
An ordinary disk file mapped for sharing has filesystem persistence instead. Its name and stored bytes can survive reboot, but crash durability requires an explicit writeback design. A live mutex stored inside it is not automatically reusable after a reboot.
Prefer POSIX IPC for many new local designs because its open, close and unlink model is simpler. System V offers features POSIX does not, including atomic operations across a semaphore set. Choose for requirements, rather than treating age as proof of unsuitability.
Send complete messages through a POSIX queue
2 min readmq_open() from <mqueue.h> returns a queue descriptor of type mqd_t, or (mqd_t)-1 on failure. Use O_CREAT | O_EXCL when this process must create a fresh queue. Add O_RDONLY, O_WRONLY or O_RDWR for the required direction. A creator supplies permissions and a struct mq_attr.
mq_maxmsg and mq_msgsize set capacity and maximum message bytes at creation. mq_curmsgs reports the queued count. mq_flags reports nonblocking mode for the open queue description. Creation attributes do not resize existing queues. mq_setattr() changes O_NONBLOCK, not capacity.
Save this original example as mq_priority.c. It queues two messages before receiving either, so priority can affect their order. Its small limits are requests, not promises about every target's configuration.
#define _POSIX_C_SOURCE 200809L
#include <fcntl.h>
#include <mqueue.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char **argv)
{
if (argc != 2) {
fprintf(stderr, "usage: %s /queue-name\n", argv[0]);
return EXIT_FAILURE;
}
struct mq_attr requested = { .mq_maxmsg = 4, .mq_msgsize = 64 };
mqd_t q = mq_open(argv[1], O_CREAT | O_EXCL | O_RDWR,
0600, &requested);
if (q == (mqd_t)-1) {
perror("mq_open");
return EXIT_FAILURE;
}
int status = EXIT_FAILURE;
if (mq_send(q, "sample", 6, 1) == -1 ||
mq_send(q, "alarm", 5, 7) == -1) {
perror("mq_send");
goto cleanup;
}
for (int i = 0; i < 2; ++i) {
char buffer[64];
unsigned int priority;
ssize_t n = mq_receive(q, buffer, sizeof buffer, &priority);
if (n == -1) {
perror("mq_receive");
goto cleanup;
}
printf("priority=%u bytes=%zd data=%.*s\n",
priority, n, (int)n, buffer);
}
status = EXIT_SUCCESS;
cleanup:
if (mq_close(q) == -1) { perror("mq_close"); status = EXIT_FAILURE; }
if (mq_unlink(argv[1]) == -1) { perror("mq_unlink"); status = EXIT_FAILURE; }
return status;
}
Compile on Linux and give the run a name containing your shell's process ID. -lrt supports toolchains that place these interfaces in the real-time library. Keep it for older embedded C libraries even if your host links without it.
cc -std=c11 -O2 -Wall -Wextra mq_priority.c -o mq_priority -lrt
./mq_priority "/learner-mq-$$"
The program prints priority, byte count and text. The waiting alarm arrives before the sample. Equal priorities preserve send order. Priorities range from zero through sysconf(_SC_MQ_PRIO_MAX) - 1; message priority does not set process scheduling priority.
mq_receive() needs a buffer at least as large as the queue's configured mq_msgsize, even for a short pending message. It returns bytes, not a C string. Use mq_getattr() when opening someone else's queue. Validate received lengths before interpreting a binary payload. mq_receive(3) specifies these buffer rules.
A blocking send waits when full; a blocking receive waits when empty. O_NONBLOCK changes these cases to EAGAIN. EMSGSIZE means a send is too large or a receive buffer too small. Interrupted waits can return EINTR; retry only while your shutdown policy allows it. Timed operations use an absolute real-time clock deadline, so clock changes matter.
For chat, use one queue per direction. Multiple readers compete; queues do not broadcast. Sustained urgent traffic can starve routine messages. Reserve capacity or another path for shutdown and control.
Create shared memory without publishing half-built state
3 min readA mapping connects virtual addresses to underlying memory or file storage. shm_open() opens POSIX shared storage. ftruncate() sets its size. mmap() makes its bytes accessible.
1Select exactly one creator
The creator uses shm_open(name, O_CREAT | O_EXCL | O_RDWR, 0600). A new object starts with length zero. The 0600 mode grants its owner read and write access, subject to umask, which removes requested permission bits. Group sharing needs intentional ownership and mode settings, not a world-writable shortcut.
2Size and map the object
This creator-side function returns a mapping or MAP_FAILED. Its caller initializes the contents, then eventually calls munmap() and shm_unlink(). Clients must wait for initialization.
#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
void *create_region(const char *name, size_t bytes)
{
/* This teaching interface accepts only these small region sizes. */
if (bytes == 0 || bytes > 65536) {
errno = EINVAL;
return MAP_FAILED;
}
int fd = shm_open(name, O_CREAT | O_EXCL | O_RDWR, 0600);
if (fd == -1)
return MAP_FAILED;
void *base = MAP_FAILED;
if (ftruncate(fd, (off_t)bytes) == 0)
base = mmap(NULL, bytes, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
int saved_errno = errno;
close(fd);
if (base == MAP_FAILED) {
shm_unlink(name);
errno = saved_errno;
}
return base;
}
MAP_SHARED makes changes shared. MAP_PRIVATE creates private changes and is wrong for this purpose. Closing the descriptor after a successful mapping does not remove the mapping. An attaching client opens without O_TRUNC, checks the size with fstat(), then maps the agreed length. See shm_overview(7).
3Publish readiness after initialization
Initialize headers, indices and synchronization objects before clients use the region. A discoverable name does not prove readiness. Start simply: initialize before fork(). For unrelated processes, use a startup handshake or a supervisor that delays clients until initialization succeeds.
Claim creation: Create exclusively and record responsibility for cleanup.
Allocate storage: Set the full object size before any client accesses bytes.
Prepare contents: Initialize the layout and process-shared synchronization exactly once.
Announce readiness: Complete a separate handshake before consumers inspect the region.
Validate attachment: Check size, layout version and access permissions before use.
Retire safely: Stop participants, release mappings and remove the owned name.
4Define bytes both processes understand
An application binary interface, or ABI, specifies low-level layout and calling conventions. C structure padding and long width may differ between 32-bit and 64-bit programs. Store offsets from the mapping base, not raw pointers. Check offset bounds before use. Keep native mutex objects within a compatible C-library and architecture environment.
You can also open() an ordinary file, size it, and map it with MAP_SHARED. That shares bytes too. msync() concerns backing-file writeback; it does not replace locks. Access beyond the backing object's end can cause SIGBUS, a signal reporting an invalid backing-memory access. Another process truncating a live object can trigger the same failure.
Transfer ring slots with process-shared synchronization
4 min readA semaphore supports sem_wait() to take one permit and sem_post() to return one. Successful operations also provide the synchronization needed for the associated data handoff. The previous lesson used them within a process; now their storage and initialization must support different processes.
A named semaphore uses <semaphore.h> and, for N initial permits, sem_open("/learner_slots", O_CREAT | O_EXCL, 0600, N). Check against SEM_FAILED, not NULL. Release your handle with sem_close() and its name with sem_unlink(). Opening an existing semaphore preserves its count.
An unnamed semaphore lives in storage you provide. Use sem_init(®ion->items, 1, 0) for interprocess sharing. The nonzero second argument requests process sharing, and region must actually be mapped shared. Setting it on a stack variable does not make that stack shared. Use sem_destroy() only after all users and waiters have stopped. sem_overview(7) distinguishes both forms.
1Give each permit a meaning
For capacity N, initialize space permits to N and item permits to zero. A bounded ring has producer and consumer positions that wrap around. With one producer and one consumer (SPSC), each position has one writer. Semaphores transfer permission before slot access.
2Publish and reclaim slots
The producer waits for space, fills a slot, advances, then posts an item. The consumer waits for an item, reads, advances, then posts space. This sleeping alternative to atomic SPSC needs its own correctness argument.
This helper resumes after an unrelated signal and returns other errors.
#include <errno.h>
#include <semaphore.h>
int wait_permit(sem_t *semaphore)
{
int rc;
do {
rc = sem_wait(semaphore);
} while (rc == -1 && errno == EINTR);
return rc;
}
3Add exclusion when ownership is shared
With several producers, permit counts alone cannot stop two producers choosing the same write index. Protect index allocation and slot publication with an appropriate mutex or redesign the queue. A mutex is a lock owned by the thread that acquires it; it protects a group of related accesses.
Initialize mutex attributes with pthread_mutexattr_init(). Select PTHREAD_PROCESS_SHARED with pthread_mutexattr_setpshared(), then initialize the mapped mutex using pthread_mutex_init(). Destroy the attributes afterward with pthread_mutexattr_destroy(). Check return codes; pthread functions usually return errors directly. A default mutex remains process-private even in shared storage.
Never wait for an item while holding a mutex the producer needs to create it. Establish a consistent lock order and keep critical sections, the protected portions of code, short. A named semaphore used as a binary lock starts at one, but has no owner enforcement or automatic owner-death recovery.
Going deeperA crashed participant can strand a correct ring
Suppose a producer takes a space permit and dies before posting an item. The ring loses usable capacity. Ordinary POSIX semaphores do not automatically restore it.
A robust process-shared mutex, configured with PTHREAD_MUTEX_ROBUST, can report EOWNERDEAD when its owner dies. The caller then owns the lock and must repair protected data before calling pthread_mutex_consistent(). It does not repair separate semaphore counts or unfinished records. Without a credible repair rule, stop all participants and create a fresh generation. See robust mutex attributes.
POSIX synchronization supplies the ordering for ordinary shared CPU memory on supported Linux targets. Volatile does not replace it. Device registers and DMA buffers have different rules; mapping normal shared memory does not create a device-coherency protocol.
Recognize System V message queues and selection rules
2 min readVendor services may use System V queues through msgget(), msgsnd(), msgrcv() and msgctl(). msgget(key, IPC_CREAT | IPC_EXCL | 0600) creates one exclusively. Without creation flags, a matching key locates an existing queue.
Every message begins with a positive long type. The remaining bytes form its payload. The size argument excludes the leading type. Passing the size of the entire structure is a common out-of-bounds read or write bug.
This fragment sends four payload bytes and receives type 3. Its caller creates and cleans up the queue.
#include <sys/msg.h>
int exchange_type_three(int queue_id)
{
struct packet {
long type;
unsigned char payload[4];
} sent = { .type = 3, .payload = { 1, 2, 3, 4 } }, received;
if (msgsnd(queue_id, &sent, sizeof sent.payload, 0) == -1)
return -1;
ssize_t n = msgrcv(queue_id, &received, sizeof received.payload, 3, 0);
return n == -1 ? -1 : 0;
}
A System V message type is a positive integer used by a receiver to select eligible messages. A receive selector of zero takes the first queued message. A positive selector takes the first message with exactly that type. A negative selector takes the first message of the lowest type not exceeding its absolute value.
For queued types 5, 2 and 4, selector zero takes type 5. Selector 4 takes type 4. Selector -4 takes type 2. Each example assumes all three remain queued. A message's type does not authenticate its sender.
IPC_NOWAIT avoids sleeping. An unmatched receive returns ENOMSG; a send to a full queue returns EAGAIN. A too-small receive returns E2BIG and leaves the message queued. MSG_NOERROR instead permits truncation, discarding the omitted bytes. Avoid it for structured commands. msgop(2) defines the selection and truncation rules.
msgctl(id, IPC_STAT, &info) retrieves metadata, including permissions and byte limits. msgctl(id, IPC_RMID, NULL) removes the queue and discards queued messages. Blocked operations wake with failure, commonly EIDRM. Unlike POSIX unlink, it does not preserve a usable queue for existing clients.
Inspect with ipcs -q; use ipcs -q -i ID for one queue, replacing ID with its listed identifier. ipcrm -q ID removes it. These util-linux commands inspect System V IPC, not POSIX queues. Remove only objects you own and have deliberately retired.
Understand System V shared memory and semaphore sets
3 min readSystem V shared memory uses shmget() to create or find a segment, a shared storage region. shmat(id, NULL, 0) attaches it at a kernel-chosen address. Its failure value is (void *)-1, not NULL. shmdt(address) detaches that mapping.
shmctl(id, IPC_RMID, NULL) marks a segment for destruction. Existing attachments keep it alive until the last detach. Linux permits attaching an already marked segment while it still exists, but portable code must not depend on this. Arrange attachment before removal. shmctl(2) documents this special lifetime rule.
Like POSIX mappings, these segments need synchronization and offset-based layouts. ipcs -m lists them. Detaching all processes does not normally remove an unmarked segment. A crashed creator can leave memory allocated for the next service start to discover.
A System V semaphore set contains several indexed semaphore counters in one IPC object. semget(key, count, flags) creates or locates a set. semctl() initializes members with SETVAL or SETALL, reads metadata, and removes sets with IPC_RMID. Initialize explicitly; portable startup must not assume Linux's zero initialization.
semop() accepts an array of struct sembuf. Each entry names a semaphore index, an adjustment and flags. A positive adjustment adds permits. A negative adjustment waits until enough permits exist, then subtracts them. A zero adjustment waits for the selected value to become zero. The entire array succeeds together or makes none of its changes.
This fragment reserves one permit from each of two initialized members. The meaning of those permits determines whether undo is appropriate.
#include <sys/sem.h>
int reserve_pair(int set_id)
{
struct sembuf operations[2] = {
{ .sem_num = 0, .sem_op = -1, .sem_flg = 0 },
{ .sem_num = 1, .sem_op = -1, .sem_flg = 0 }
};
return semop(set_id, operations, 2);
}
IPC_NOWAIT on an unsatisfiable operation returns EAGAIN. Signals can interrupt a wait with EINTR; semop() is not automatically restarted by SA_RESTART. An atomic grouped operation avoids partial reservation, but does not promise fairness among different competing requests. POSIX semaphores are individual objects and lack this grouped-operation interface.
Going deeperUse SEM_UNDO only when its accounting matches ownership
SEM_UNDO records a reverse adjustment for the calling process. For a binary lock, use it on both the decrement that acquires and the increment that releases. Normal release cancels the recorded adjustment; process exit after an unreleased acquisition can restore the permit.
It cannot undo bytes already written to shared memory. It is also a poor automatic fix for producer-consumer item counts: one process posts items that another consumes. Restoring a producer's posts on exit would not match ownership. Undo adjustments are not inherited across fork() but survive exec(). Linux threads normally share undo bookkeeping. See semop(2).
Use ipcs -s for sets and ipcrm -s ID for deliberate removal. The /proc/sysvipc/msg, /proc/sysvipc/sem and /proc/sysvipc/shm files expose corresponding tables. These objects normally survive the creator's exit. Record an owner and cleanup plan for each.
Treat eventfd as a counter with a lifetime
2 min readeventfd(0, EFD_NONBLOCK | EFD_CLOEXEC) creates a counter initially zero. Include <sys/eventfd.h>. EFD_CLOEXEC closes that descriptor when a successful exec() replaces the program.
Use the public C-library function eventfd(). At the system-call level, Linux has the original eventfd call and eventfd2, which accepts flags. A trace can show eventfd2 although your source calls eventfd(). There is no ordinary application requirement to call a separate public eventfd2() function.
Pass a uint64_t to write(). A successful eight-byte write adds its value. Without EFD_SEMAPHORE, a successful read returns the accumulated value and resets the counter to zero. With that flag, each read returns one and subtracts one. A zero counter causes blocking, or EAGAIN in nonblocking mode.
Always request exactly sizeof(uint64_t) bytes. Writes of another size fail, as do read buffers smaller than eight bytes. The maximum writable counter value is UINT64_MAX - 1; writing UINT64_MAX is invalid. Addition beyond capacity blocks or returns EAGAIN. Values use host byte order. See eventfd(2).
The object has no reopenable application name. Descriptors inherited through fork() share its counter. Unrelated processes need descriptor transfer through a Unix-domain socket, covered next session. Copying a descriptor's integer does not transfer the object.
The final reference disappearing releases the counter. Leaked descriptors can keep it alive and complicate shutdown. Closing a descriptor from another thread is not a reliable cancellation protocol for a blocked operation. Send an explicit shutdown notification, join the worker, then close its resources.
Going deeperFollow the kernel counter and wakeup path
In Linux's fs/eventfd.c, struct eventfd_ctx holds a counter and a wait queue. A wait queue records tasks interested in a condition. The implementation protects counter changes with a kernel lock and wakes interested waiters when readiness changes.
This lock protects the counter, not your shared ring. Kernel serialization of notifications does not define which user-space slot is safe to read. Keep the ring's mutex, semaphore or proven atomic publication rules. For POSIX queues, ipc/mqueue.c instead manages priority groups and waiting senders and receivers, including direct handoff opportunities.
A practical producer writes the shared work first, publishes it under the queue's synchronization, then writes one to eventfd. Reversing publication and notification lets the consumer wake before work is ready. The count is a hint to inspect work, unless your protocol carefully maintains an exact one-to-one relationship.
Combine notifications with an event loop
4 min readAn event loop waits, handles work, then waits again. Linux epoll monitors file descriptors. Here you need creation, registration and waiting; full coverage comes later. EPOLLIN requests readable events. Default level triggering keeps reporting readiness while the readable condition remains true.
In event_wake.c, a worker wakes the main thread without transferring payload data. Descriptors remain open until the worker finishes.
#define _GNU_SOURCE
#include <errno.h>
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>
#include <sys/eventfd.h>
#include <unistd.h>
static void fail(const char *where)
{
perror(where);
exit(EXIT_FAILURE);
}
static void *notify_main(void *argument)
{
int fd = *(int *)argument;
uint64_t one = 1;
ssize_t n;
do { n = write(fd, &one, sizeof one); } while (n == -1 && errno == EINTR);
if (n != (ssize_t)sizeof one)
fail("eventfd write");
return NULL;
}
int main(void)
{
int wake = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
if (wake == -1) fail("eventfd");
int loop = epoll_create1(EPOLL_CLOEXEC);
if (loop == -1) fail("epoll_create1");
struct epoll_event interest = { .events = EPOLLIN, .data.fd = wake };
if (epoll_ctl(loop, EPOLL_CTL_ADD, wake, &interest) == -1)
fail("epoll_ctl");
pthread_t worker;
int error = pthread_create(&worker, NULL, notify_main, &wake);
if (error) {
fprintf(stderr, "pthread_create: %s\n", strerror(error));
return EXIT_FAILURE;
}
struct epoll_event ready;
int count;
do { count = epoll_wait(loop, &ready, 1, -1); }
while (count == -1 && errno == EINTR);
if (count == -1) fail("epoll_wait");
for (;;) {
uint64_t value;
ssize_t n = read(wake, &value, sizeof value);
if (n == (ssize_t)sizeof value) {
printf("wake count=%llu\n", (unsigned long long)value);
continue;
}
if (n == -1 && errno == EINTR) continue;
if (n == -1 && errno == EAGAIN) break;
fail("eventfd read");
}
error = pthread_join(worker, NULL);
if (error) {
fprintf(stderr, "pthread_join: %s\n", strerror(error));
return EXIT_FAILURE;
}
if (close(loop) == -1) fail("close epoll");
if (close(wake) == -1) fail("close eventfd");
return EXIT_SUCCESS;
}
The worker writes once; the main thread waits and drains until EAGAIN. An earlier write would also remain counted. Compile and run to observe the count.
cc -std=c11 -O2 -Wall -Wextra -pthread event_wake.c -o event_wake
./event_wake
For a real shared queue, drain the notification counter, then inspect and consume published queue entries under their synchronization. If publication follows your final empty check, the producer's later eventfd write makes the descriptor readable again. If publication happens earlier, you consume the entry now or receive a harmless extra wakeup later.
Reserve storage: The producer obtains a slot through the queue's own protocol.
Publish work: It fills the payload and makes the entry visible safely.
Count notification: It writes one to the shared eventfd.
Observe readiness: epoll reports the eventfd readable to the consumer.
Drain and inspect: The consumer clears notifications and consumes published work.
Wait again: A later notification remains counted even across the wait boundary.
Always-notify is easier to prove than signaling only when a ring appears empty. Suppressing writes needs an explicit sleep-state handshake to avoid a lost wakeup, where work remains available but its consumer sleeps. Under a coalesced-wakeup policy, eventfd write EAGAIN means a notification is already pending. Under exact permit accounting, dropping that increment is incorrect.
POSIX queues have another asynchronous interface, mq_notify(). SIGEV_SIGNAL sends a chosen signal; block it before registration and before creating other threads. Consume it with sigwaitinfo() outside a signal handler. SIGEV_THREAD invokes a callback as a new thread would, requiring safe callback lifetimes and serialization. SIGEV_NONE registers without delivering a notification; it is not the same as unregistering with mq_notify(q, NULL).
Notification is one-shot and only concerns an empty-to-nonempty transition. One process may register; a competing registration can fail with EBUSY. A blocked receiver takes precedence. Register before your initial nonblocking drain. After a notification, re-register before draining again, then wait for the next notification. This avoids an arrival between draining and registration leaving work unnoticed. See mq_notify(3).
Linux queue descriptors can also participate directly in epoll; that is Linux-specific, not portable POSIX behavior. Choose one consumption strategy. Independent callback and event-loop consumers make message ownership harder to reason about.
Choose deliberately and diagnose common mistakes
4 min readPersistence here describes object lifetime, not automatic message recovery or durable command processing.
| Mechanism | Lifetime | Payload and size | Notification and control |
|---|---|---|---|
| POSIX queue | Until unlink and final close, or namespace teardown | Bounded discrete messages, configured maximum | Blocking waits, priorities, optional arrival notification |
| Shared memory | Named object persists; mappings hold references | Region sized by application and resource limits | Add locks, semaphores or eventfd |
| Pipe or FIFO | Buffered data tied to open endpoints; FIFO pathname may persist | Bounded byte stream, application framing | End-of-file after buffers drain and all writers close |
| Unix socket | Endpoint lifetime tied to references; pathname may need removal | Stream or messages by socket type | Readiness, peer connection state, descriptor passing |
Choose POSIX queues for short prioritized commands, shared memory for large frames with known ownership, and pipes for parent-child streams. Consider Unix sockets for connections and descriptor exchange.
Linux memfd_create() is useful when discovery by public name is unnecessary. Define _GNU_SOURCE, include <sys/mman.h>, and call memfd_create("learner-frame", MFD_CLOEXEC). Size the returned descriptor with ftruncate() and map it shared. Its descriptive string is not a rendezvous name. Share the descriptor through inheritance or later Unix-socket descriptor passing. The object disappears after its final reference is released.
With MFD_ALLOW_SEALING, file seals can restrict later changes, such as shrinking. Seals are restrictions applied through fcntl(); they do not lock payload access. A shrink seal can prevent a peer from invalidating your mapping by truncation. See memfd_create(2) for the supported operations.
1Match the symptom to the object
| Symptom | Likely cause | Diagnose | Fix |
|---|---|---|---|
mq_open reports EINVAL |
Invalid requested limits | Compare attributes with mqueue limits | Request supported bounds |
Restart reports EEXIST |
Stale object or live owner | Inspect ownership and service state | Reconnect intentionally or retire the old generation |
| Shared state differs | Private mapping or different objects | Compare mapping paths and creation history | Map the same object with MAP_SHARED |
| Wait never completes | Lost permit, wrong sharing, dead peer | Trace waits and inspect all participants | Correct initialization and add failure handling |
Mapping access raises SIGBUS |
Truncation or unavailable backing space | Check size, writers and tmpfs space | Stop truncation and enforce a memory budget |
| Callback stops arriving | One-shot notification not rearmed | Trace registration and receive order | Rearm before draining |
2Inspect limits and ownership
Read limits rather than memorizing host defaults. These commands show POSIX queue bounds and System V memory and message bounds. Missing files on a minimal image can indicate disabled kernel support or unavailable procfs.
cat /proc/sys/fs/mqueue/msg_max /proc/sys/fs/mqueue/msgsize_max
cat /proc/sys/fs/mqueue/queues_max
cat /proc/sys/kernel/shmmax /proc/sys/kernel/shmall
cat /proc/sys/kernel/msgmax /proc/sys/kernel/msgmnb /proc/sys/kernel/msgmni
ls -ld /dev/shm /dev/mqueue
df -h /dev/shm
ipcs -q
ipcs -m
ipcs -s
shmmax bounds one System V segment in bytes. shmall limits aggregate System V shared memory in pages, not bytes. msgmax limits a System V message, msgmnb sets default queue byte capacity, and msgmni limits queue count. These are separate from POSIX mqueue controls. Their definitions appear in the kernel sysctl documentation.
Also inspect RLIMIT_MSGQUEUE, the per-real-user POSIX queue memory budget, and descriptor limits for the process. Queue metadata consumes resources beyond payload bytes. A small /dev/shm mount can constrain POSIX memory independently of shmmax.
3Trace the failing operation
Trace creation, transfer and cleanup; -f follows created tasks. Expect eventfd2 for eventfd and possibly mq_timedsend or mq_timedreceive for queue transfers.
strace -f -o mq.trace ./mq_priority "/learner-trace-$$"
strace -f -o wake.trace ./event_wake
Find the first failed call, then where progress stops. Inspect /proc/PID/maps and /proc/PID/fd using actual process IDs. Compare both views. Containers can separate IPC namespaces or /dev/shm mounts despite matching names. System security policy can restrict access beyond ordinary mode bits.
Cross-compilation needs the target C library's headers and libraries. BusyBox applets vary by build; ipcs, ipcrm and strace may need separate packages. Check kernel options such as CONFIG_SYSVIPC, CONFIG_POSIX_MQUEUE and CONFIG_EVENTFD. Host success does not prove the target image supports every mechanism.
Measure latency, throughput, CPU time and overload behavior on realistic payloads. Include mapping faults, cache contention and participant crashes. Next lesson, place IPC startup and cleanup within a service manager's lifecycle.
Try it
1 min readOpen the lesson lab and practice pack for a two-process priority chat, a named-semaphore shared ring, and an eventfd-driven epoll exercise. Predict message order before running the chat, then explain how backpressure changes it. For the ring, document creator election, slot ownership and recovery after a participant dies. Repeat the wakeup exercise on your ARM target if available, and bring a short trace plus a cleanup plan to the next lesson on daemons and systemd services.
Key takeaways
- Delivery, storage and notification are separate jobs, even when one API combines them.
- Closing a handle does not unlink a persistent named IPC object.
- POSIX queues preserve messages and deliver higher priorities before lower ones.
- Shared mappings need an initialization protocol and interprocess synchronization.
- System V message types select messages; semaphore sets support atomic grouped operations.
- SEM_UNDO adjusts semaphore counts after exit but cannot repair shared application data.
- eventfd counts notifications; the shared queue remains the source of truth for work.
- Measure on the target and diagnose limits before increasing resource budgets.
Reflect
- Explain to a colleague, in two sentences: identify POSIX and System V IPC objects and explain their naming and lifetime rules.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Official docsLinux kernel IPC limits
- Man pagemq_overview(7)
- Man pagemq_notify(3)
- Man pagemsgop(2)
- Man pageshm_overview(7)
- Man pagesem_overview(7)
- Man pagesemop(2)
- Man pageeventfd(2)
- Man pagememfd_create(2)
Official documentation first. Links checked when the lesson was published.