Why addresses are virtual
2 min readWhen you compile a program and it prints the address of a variable, that address is a an address in the private space a process sees, which the hardware translates to a physical location before memory is touched. The memory management unit, the part of the CPU that translates virtual addresses into physical addresses and enforces permissions on each access is the hardware that does the translation. The physical address is the real location on the RAM chip itself. The two rarely match.
A small example makes the idea concrete. Start two copies of a program that prints the address of a global variable. Both copies print the same value, perhaps 0x404010. They cannot both own that physical byte, so the MMU maps each copy's 0x404010 to a different page of physical RAM. Each program believes it owns a whole address space starting at zero; the hardware keeps them apart behind the scenes.
This indirection buys three things. First, isolation: one process cannot read or write another process's memory, because its page tables simply contain no mapping to it. Second, protection: an access to an unmapped address, or a write to a read-only one, stops the CPU and the kernel kills the program with SIGSEGV instead of corrupting the machine. Third, relocation: the same code or library can be loaded at any physical location and mapped wherever the program expects it, and several processes can share one physical copy of a library while each sees it at its own address.
The embedded contrast is stark. A Cortex-M microcontroller has no MMU, so every running task sees the one flat physical address space. A bad pointer in one task can silently overwrite the operating system itself. Linux requires an MMU, which is exactly why it runs on Cortex-A parts such as the STM32MP1's Cortex-A7 and the Raspberry Pi's cores, and not on Cortex-M. The MMU is the foundation of every guarantee Linux makes about memory.
Why caches exist
2 min readA modern core can retire an instruction in a fraction of a nanosecond, but reading one word from external DDR RAM can take tens of nanoseconds, and the first miss after a cold start can take hundreds. Without caches, the core would spend most of its time waiting. The gap between core speed and memory speed has widened for decades, so today's systems stack several levels of cache between the core and RAM.
The hierarchy runs from small and fast to large and slow. Registers sit inside the core. Level 1 (L1) cache is next, typically a few tens of kilobytes, split into an instruction cache and a data cache so a fetch and a load can happen together. Level 2 (L2) is larger, from 256 KB to over 1 MB, and is unified, holding instructions and data together. Some big systems add an L3 shared by all cores. Below the caches sits RAM, and below that, flash storage. Each level is bigger and slower than the one above it.
Caches move data in fixed-size blocks called a the fixed block, usually 32 or 64 bytes, that a cache reads from and writes to memory as one unit. When the core reads one byte, the cache fetches the whole surrounding line. This works because of two patterns real programs show. Temporal locality: a value just used is likely to be used again soon. Spatial locality: a value near one just used is likely to be used next, because code runs in sequence and data sits in arrays and structs.
When the core finds what it wants in a cache, that is a hit; otherwise it is a miss and the line is fetched from the next level down. The fraction of accesses that hit, the hit rate, decides most of the system's effective speed. On embedded parts the sizes are small: a Cortex-A7 has 32 KB of L1 instruction and 32 KB of L1 data cache and a shared 256 KB L2, while most Cortex-M microcontrollers have no cache at all. Fewer transistors mean less area and less power, which is why cache sizes are a big part of an SoC's silicon budget.
Page tables: how the MMU translates an address
4 min readMemory is divided into pages, fixed blocks usually 4 KiB (4096 bytes) on both 32-bit Arm and AArch64 Linux. The the tree of tables the MMU walks, whose entries map a virtual page to a physical page and carry its permissions and memory attributes is the structure that holds the mappings. Storing one entry per page as a flat list would be wasteful, because a process uses only a small slice of its address space. Instead, Linux uses a multi-level tree so that unused parts of the address space cost no table entries at all.
On AArch64 with the default 4 KiB pages and a 48-bit virtual address, there are four levels. The virtual address is split into five fields: a 9-bit index into a level 0 table, a 9-bit index into level 1, 9 bits for level 2, 9 bits for level 3, and the low 12 bits as the offset within the page. Each table holds 512 entries of 8 bytes. The walk starts from a base register that points at the level 0 table, and each entry either points at the next-level table or, at the last level, at the physical page itself.
Split the address: the MMU reads the five fields from the 48-bit virtual address: four 9-bit indexes and a 12-bit offset.
Read the base register: the table base register, TTBR0_EL1 for user code or TTBR1_EL1 for the kernel, points at the level 0 table.
Walk level 0: the level 0 index picks an entry; its address field points at the level 1 table.
Walk level 1 and level 2: the next two indexes pick entries that point at the following tables.
Reach level 3: the last index picks a page entry holding the physical page number and its attributes.
Combine: the MMU joins the physical page number with the low 12 bits of the virtual address to form the physical address.
Each table entry also carries permissions and attributes alongside the address: whether the page is readable, writable and executable, whether it is user or kernel memory, and how the cache and the rest of the system must treat it. This is what makes protection possible: the MMU checks every access against the entry, and a violation raises an exception the kernel turns into a SIGSEGV. Linux stores a separate tree of tables for each process, so switching processes means pointing the base register at a different tree, which instantly changes the whole visible world.
Walking four tables on every access would be far too slow, so the MMU keeps a translation lookaside buffer, a small per-core cache of recently used virtual-to-physical translations that makes repeat accesses skip the table walk. Most accesses hit the TLB and never touch the tables. When a process is switched out, its translations must be invalidated, or a fast flush avoided by tagging each entry with an address space identifier (ASID) that distinguishes one process from another; entries for a different ASID simply do not match.
Going deeperGranules, 52-bit addresses and the kernel's second base register
AArch64 supports three translation granules: 4 KiB, 16 KiB and 64 KiB, chosen at boot by the kernel. The 16 KiB granule needs only three levels, and 64 KiB only two. Larger granules mean fewer levels and a bigger reach per table, but more wasted space at the edges of a mapping. The 48-bit address size can be extended to 52 bits with 64 KiB granules, and Linux calls this LPA. The two base registers also matter: TTBR0_EL1 translates the lower half of the address space used by user programs, while TTBR1_EL1 translates the upper half where the kernel lives. Splitting the space in two lets the kernel keep its own mappings in every process without sharing user tables, and it is why the next section talks about a user half and a kernel half.
The kernel's own address space
3 min readWhen Linux runs, it reserves a large block of the virtual address space for itself and gives the rest to whichever process is running. User addresses map to different physical pages in each process. Kernel addresses map to the same physical memory no matter which process is running, because there is one kernel and its code and data never change identity.
The 32-bit Arm layout is the simpler one to read. With the default split, user space owns the lower 3 GB, from 0x00000000 to 0xBFFFFFFF, and the kernel owns the top 1 GB, from 0xC0000000 to 0xFFFFFFFF. The boundary is a compile-time constant called PAGE_OFFSET. On a 64-bit AArch64 system the numbers change but the idea does not: the 48-bit address space is cut into two 256 TB halves, the lower half translated through TTBR0_EL1 for user code and the upper half through TTBR1_EL1 for the kernel.
The kernel half is itself carved into regions, each with a job. The lowmem (on 32-bit) or linear map (on AArch64) is a direct mapping of physical RAM at a fixed offset, so the kernel can reach any physical page by simple arithmetic. The vmalloc region provides virtually contiguous memory for drivers that need large or scattered allocations and for remapping device registers. The vmemmap region holds the kernel's array of per-page bookkeeping, and a small module region holds loaded kernel modules. The boot log prints this layout on many systems.
# typical output from a 32-bit ARM kernel at boot (yours will differ)
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
vmalloc : 0xe0800000 - 0xff000000 ( 488 MB)
lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
modules : 0xbf800000 - 0xbfe00000 ( 6 MB)
.text : 0xc0008000 - 0xc0763c90 (7536 kB)
.data : 0xc07a0000 - 0xc0827240 ( 541 kB)
.bss : 0xc0827240 - 0xc089e940 ( 478 kB)
The point of reading this log is the boundary: everything below 0xC0000000 belongs to user programs, everything at and above it belongs to the kernel. The kernel's own code (.text), its read-only data, its variables (.data) and its zero-initialised area (.bss) all live in lowmem. You can see the same boundary from the other side by looking at any process's memory map with /proc/PID/maps, where user mappings always sit below the kernel line.
One consequence matters for debugging on the board: kernel memory is not demand-paged the way user memory is. A kmalloc allocation has real physical memory behind it immediately, kernel pages are never swapped out, and the kernel's mappings are the same for every process. That is what makes the kernel a fixed, shared resident, and it is why kernel and user code need different rules, a theme the next section deepens.
The SoC memory map and memory-mapped I/O
3 min readA desktop CPU talks to its peripherals over buses that hide most of the detail. An SoC puts everything in one flat physical address map. A range of addresses is real RAM; other ranges are memory-mapped I/O, the technique of giving each peripheral's control registers ordinary addresses in the same physical space as RAM, so the CPU accesses them with loads and stores. Reading a register at one address returns a device's status; writing to another changes its behaviour. The CPU has no idea which is which; the distinction lives only in the memory map, the SoC's fixed layout of which address means what.
The STM32MP1 shows the pattern. DDR RAM sits at 0xC0000000, 512 MB on the Discovery kit. Peripheral registers occupy a wide band around 0x40000000 and above, where each UART, MMC controller and timer has its own small window. A block of on-chip SYSRAM sits at 0x2FFC0000, and the boot ROM at the very bottom, 0x00000000. Nothing is physically "at" these addresses the way RAM is; the addresses are just the labels the bus uses to route an access to the right block.
Linux exposes this map in /proc/iomem, which lists the physical ranges the kernel knows about and which driver claims each one.
# typical /proc/iomem on an STM32MP1-class board (yours will differ)
$ cat /proc/iomem
00000000-0001ffff : Boot ROM
10000000-1004ffff : SRAM
2ffc0000-2fffffff : SYSRAM
40000000-5fffffff : Peripheral bus
c0000000-dfffffff : System RAM
Each range is a claim: System RAM is where the allocator hands out pages, while the peripheral band is subdivided by drivers as they probe, so you will see entries such as serial@40010000 appear under the bus once the UART driver loads. The companion file /proc/ioports lists a different thing entirely, the legacy port I/O space used by x86's IN and OUT instructions; on an Arm SoC it is usually empty, because Arm has no port I/O and everything is MMIO.
This is why memory attributes matter. RAM is normal memory: it can be cached, reads can be speculated, and accesses can be reordered, because a load or store to RAM has no hidden effect. A peripheral register is the memory type the MMU assigns to peripheral registers, which forbids caching, speculation and reordering because each access has a hardware side effect. If a driver mapped a status register as normal cacheable memory, the core might read it once, keep the stale value in cache, and never see the hardware change again. Device memory forbids that, and it also stops the compiler and CPU from merging or reordering the accesses.
The kernel gives you the correct accessors so you never map device memory as normal memory by accident. For user memory you use get_user and put_user, which safely copy single values between the kernel and a user process and handle faults on the way. For device registers you use readl and writel (or ioread32 and iowrite32), which go through a region mapped with device attributes and emit exactly one access with the right ordering. A plain C pointer dereference is wrong for both: it has no fault handling for user memory and no device semantics for registers.
On some devices, reading a register clears an interrupt or advances a FIFO. That is why a driver must never read a register speculatively or cache its value, and why the debug tool devmem can visibly disturb hardware when you peek at registers during bring-up. Only poke a register when you mean it.
Caches in depth: policies, coherency and DMA
4 min readWhen the core writes, the cache has two choices. A write-back cache stores the new value only in the cache and marks the line dirty, writing it to RAM only when the line is evicted or explicitly cleaned. A write-through cache writes to the cache and to RAM at once, so RAM is always current but every write pays the slow path. Almost all normal RAM on Linux systems is mapped write-back, because it is faster and uses far less memory bandwidth; write-through is used only in special cases, such as frame buffers that hardware must observe promptly.
The remaining subtleties follow from that choice. A write-back cache must record which lines are dirty, and a dirty line must be written out, or cleaned, before the underlying RAM is considered correct. A cache can also be write-allocate, pulling a line in on a write miss, or no-write-allocate, writing straight through to memory without filling the cache. Each combination trades speed against traffic.
Coherency is the problem that appears the moment there is more than one cache. Two cores each have their own L1, so a write on one core can leave the other core's L1 holding a stale copy. Hardware keeps them consistent with a protocol such as MESI, where each line is in one of a few states such as Modified, Exclusive, Shared or Invalid, and the caches snoop each other's traffic so that a read of a modified line gets the fresh value. On a Cortex-A cluster this is coordinated by a Snoop Control Unit, and the interconnect extends the same idea between clusters and to L2. The memory attributes you saw earlier carry a shareability field, which tells the hardware which other cores and caches must stay coherent for a given address; normal memory is inner-shareable, so all cores in the cluster see one coherent view.
DMA breaks the coherency assumptions. A DMA engine moves data between a peripheral and RAM without going through the CPU's caches, so on many Arm systems the caches are not coherent with DMA by default. A driver must then maintain the caches by hand around each transfer. Before the device reads a buffer, the driver cleans the relevant lines, pushing dirty data to RAM; after the device writes a buffer, the driver invalidates the lines, so the core re-reads fresh values instead of stale cache. The kernel packages this into mapping calls.
/* non-coherent DMA: map the buffer, let the device use it, then unmap */
dma_addr_t addr = dma_map_single(dev, buf, size, DMA_TO_DEVICE);
if (dma_mapping_error(dev, addr)) {
/* handle the error */
return;
}
/* the device now reads the buffer at addr; the map call cleaned the cache */
/* ... wait for the transfer ... */
dma_unmap_single(dev, addr, size, DMA_TO_DEVICE);
The direction flag tells the kernel what to do. DMA_TO_DEVICE cleans before the transfer, DMA_FROM_DEVICE invalidates after it, and DMA_BIDIRECTIONAL does both. For buffers reused many times without the CPU touching them in between, dma_alloc_coherent returns memory kept coherent with the device from the start, at a performance cost.
Going deeperCache maintenance instructions, by address and by set/way
The Arm architecture exposes cache maintenance as instructions, not as memory addresses. DC CVAC cleans a line to the point of coherency, DC CIVAC cleans and invalidates it, and DC IVAC invalidates it, each addressed by virtual address. Invalidate means "discard the cached copy", which loses data if the line was dirty, so you invalidate only when RAM is already authoritative, as after a device write. A second family, DC ISW, DC CSW and DC CISW, operates by set and way across the whole cache, which is slow and can race with normal traffic, so Linux uses it only at times such as power management when a core is being taken down. These instructions are privileged in the normal world, which is why user code never issues them and drivers reach them through the kernel's arch cache helpers.
The boot ROM and how the boot source is chosen
4 min readAt reset, a core starts fetching instructions from a fixed address. But DRAM needs a memory controller that has not been configured, and the code to configure it cannot come from DRAM. The answer is the the small, factory-written program in on-chip mask ROM that runs first at reset and loads a first-stage loader before external RAM exists. It is called mask ROM because it is etched into the silicon at manufacture: it cannot be updated, patched or erased, which is both its weakness and, as you will see, its strength.
The boot ROM works under tight constraints. It has no DRAM, so it runs from on-chip SRAM, often only a few tens of kilobytes to a little over 100 KB. It is proprietary code from the SoC vendor, with no open-source replacement. It deliberately does not configure DRAM, because DRAM timing is specific to each board's memory chip and the immutable ROM cannot know it. Its whole job is to pick a boot source, read a small first-stage loader from it into SRAM, optionally verify that loader's signature, and jump to it.
The boot source is chosen in two ways. Boot pins are external pins sampled at the instant of reset; wiring them high or low selects, for example, SD card, eMMC, QSPI flash or UART. On the STM32MP1 the BOOT pins select the source, and the ROM then tries a list of devices in a fixed order. The second mechanism is the a one-time-programmable bit in on-chip storage that permanently fixes a configuration, such as the boot source or the secure-boot key, also called eFUSE. Blown fuses cannot be unblown, so an OTP setting overrides the pins forever, which vendors use to lock a product's boot behaviour in production.
1the ROM reads the boot configuration
The ROM samples the boot pins and reads the relevant OTP fuses to learn which devices to try, and in what order.
2it loads a first-stage loader
It reads a fixed offset from the chosen device, for example the first sectors of an SD card, into on-chip SRAM. This first stage is the SPL, or on the STM32MP1, TF-A's BL2.
3it verifies, then jumps
When secure boot is on, the ROM checks the loader's signature against a hash or key burned into OTP. If it matches, the ROM jumps into SRAM; if not, boot stops.
The first-stage loader then does what the ROM could not: it configures the DRAM controller, brings external RAM up, and loads a larger second-stage loader into DRAM. This two-stage pattern, a tiny ROM-loaded stage in SRAM followed by a full bootloader in DRAM, is universal on embedded SoCs because the on-chip SRAM is too small for a full U-Boot.
The immutability of the ROM is also the foundation of secure boot. Because nothing can rewrite it, it is the one component you can always trust, the root of trust for the whole boot chain. If the ROM verifies the first stage, and that stage verifies the next, trust flows forward stage by stage. The full mechanics of that verification are taught in M4-S17B; here the point is only that the chain starts in ROM and nowhere else.
An OTP fuse cannot be reset. Blowing a fuse to force a boot source or enable secure boot is permanent, and a board with the wrong fuse set may only boot from that source ever after. Production engineers burn fuses last, on a known-good configuration, and record exactly which bits they set.
How a PC boots: BIOS and MBR, then UEFI and GPT
4 min readOn x86, after reset the processor jumps to a fixed address, 0xFFFFFFF0, near the top of its address space, and begins executing the the vendor-provided code that runs first at power-on, initialises the hardware and hands control to an operating system or bootloader stored in flash on the motherboard. Two generations of firmware exist, and they boot a PC in different ways.
The legacy firmware is the BIOS, the Basic Input/Output System, which PCs used until UEFI took over in the early 2010s (Windows 8 PCs from 2012 required it), and which many firmwares still emulated for years as a compatibility mode. The BIOS runs a power-on self-test, initialises the basic hardware, and then loads the very first sector of the chosen disk, 512 bytes, into RAM at the fixed address 0x7C00 and jumps to it. That sector is the master boot record, the first 512-byte sector of a disk, holding a small boot program, the partition table and a two-byte signature. Its layout is rigid: 446 bytes of bootstrap code, then 64 bytes holding four 16-byte partition entries, then the two signature bytes 0x55 0xAA, which the BIOS checks to confirm the sector is bootable.
The MBR's 446 bytes can do little, so it chain-loads. Its code reads the partition table, finds the active partition, and loads that partition's own boot sector, which loads the next stage, until a real bootloader such as GRUB is running and finally starts the kernel. The 32-bit LBA fields in an MBR partition entry also cap the addressable disk at 2 TiB, and the four-entry table means at most four primary partitions, worked around with extended partitions that nest logical partitions inside one slot.
The modern firmware is Unified Extensible Firmware Interface, the specification that replaced BIOS, providing boot and runtime services and loading boot applications from a dedicated partition. UEFI is a real program, usually written in C, that boots in a defined 32-bit or 64-bit flat mode rather than the BIOS's 16-bit real mode. It reads a GUID Partition Table, or GPT, instead of an MBR. GPT starts with a protective MBR in sector 0 for backward compatibility, then a GPT header and an array of partition entries, each carrying a unique GUID and a 64-bit size, with a backup copy at the end of the disk. GPT lifts the 2 TiB limit and supports far more partitions without the extended-partition hack.
A UEFI boot does not read a boot sector at a fixed address. Instead, the firmware mounts the EFI System Partition, a FAT32 partition with a well-known type GUID, and runs a boot application from it, in the fallback case the file at EFI/BOOT/BOOTX64.EFI. On a Linux system that application is usually GRUB or the systemd-boot loader, possibly behind a small shim that carries the distribution's signature for Secure Boot. Secure Boot is UEFI's signature check: firmware verifies each boot application against keys held in the firmware, refusing unsigned code.
# the UEFI boot applications on a typical Linux PC's EFI System Partition
$ ls /boot/efi/EFI
BOOT ubuntu
$ ls /boot/efi/EFI/BOOT
BOOTX64.EFI
Going deeperInside the GPT and the MBR's extended partitions
A GPT disk lays out its metadata explicitly. Logical block 0 holds a protective MBR whose single partition entry covers the whole disk so that an old tool sees one partition and does not overwrite GPT data. Logical block 1 holds the GPT header, with the disk GUID, the count and size of partition entries, and a CRC32 of the entry array. The entry array starts at logical block 2 and, with the standard 128-byte entries and 128 slots, occupies the next 32 blocks; a second copy of the header and array sits in the last blocks of the disk. Each entry stores a type GUID, a unique partition GUID, a 64-bit starting LBA and a 64-bit length. Legacy MBR disks, in contrast, extend their four entries by declaring one as an extended partition whose space is subdivided by a linked list of partition tables, each holding one logical partition and a pointer to the next.
ARM vs x86 boot, and UEFI comes to Arm
4 min readPut the two models side by side. An x86 PC has one firmware interface, BIOS or UEFI, described by a public specification, so a single operating system image can boot on any PC: the firmware discovers and describes the hardware, and the OS trusts that description. An Arm SoC has no such standard. Each vendor writes its own ROM code, which loads that vendor's first-stage loader, which loads that vendor's bootloader. The hardware is described not by firmware tables but by a device tree that the vendor ships and the bootloader hands to the kernel. That is why you cannot take a generic Linux image and expect it to boot on an arbitrary Arm board the way you can on a PC.
The Arm chain you will work with on the STM32MP1 is the concrete example, and it previews the rest of this module.
ROM code: the on-chip boot ROM reads the boot pins and loads TF-A's BL2 into on-chip SRAM.
TF-A BL2: this stage brings up DRAM and loads the remaining images into it.
TF-A BL31: the EL3 secure runtime firmware stays resident and provides services such as PSCI.
U-Boot (BL33): the normal-world bootloader takes over, reads its environment and prepares the kernel.
Kernel: U-Boot loads the kernel image and the device tree into RAM and jumps to the kernel.
init: the kernel starts the first user-space process, init, which brings up the rest of the system.
The bootloader hands the kernel three things at that boundary, and they matter later. It passes the device tree binary, which describes the hardware including the memory node; it passes the kernel command line; and it may pass an initramfs. The memory node is the part you can verify by hand: it declares the physical RAM the kernel is allowed to use, and it must match what the hardware actually has.
/* the device tree memory node declares the RAM the kernel may use */
memory@c0000000 {
device_type = "memory";
reg = <0xc0000000 0x20000000>; /* 512 MB starting at 0xC0000000 */
};
Cross-check the reg value against /proc/iomem: the System RAM range must cover exactly the region the node declares. If they disagree, the kernel either ignores real memory or tries to use memory that does not exist, and the symptom is a silent loss of RAM or a crash during early boot.
The standardisation gap is closing. Arm's SystemReady program, launched in 2020, defines firmware standards so that generic operating system images can boot on Arm hardware without per-board work, mirroring the PC model. The Server Base Boot Requirements, or SBBR, require UEFI firmware plus ACPI tables for servers. The Embedded Base Boot Requirements, or EBBR, require UEFI plus a device tree instead of ACPI, matching how embedded Arm already works. The program is split into bands: SystemReady SR for servers, SystemReady ES for embedded devices, and SystemReady IR for IoT edge hardware.
Going deeperThe SystemReady bands and why EBBR keeps the device tree
Each band pairs a hardware specification with a firmware requirement. SystemReady SR targets servers and requires SBBR, meaning UEFI with ACPI and SMBIOS, so server operating systems that already parse ACPI run unchanged. SystemReady ES targets embedded boards and requires EBBR, UEFI with the device tree, because embedded hardware has no discoverable buses and the device tree is already the universal description format there. SystemReady IR tightens EBBR for smaller IoT gateways. Certification is what makes the program real: a certified board is tested to boot stock distributions, so a vendor can advertise that a generic image works instead of shipping a custom one. For now, the boards in this course, the STM32MP1 and Raspberry Pi, still boot through the traditional ROM-and-U-Boot path rather than a SystemReady UEFI, which is exactly why learning the traditional chain matters first.
Common mistakes and how to diagnose them
3 min readStale data around DMA. Symptom: a device sends or receives bytes that are old, or data corrupts only under load. Cause: on non-coherent hardware the driver forgot the cache clean before a send, or the invalidate after a receive, so the device and the CPU read different copies. Diagnose: the bug appears only after the buffer was recently written by the CPU, and disappears when the buffer is switched to a coherent allocation. Fix: use the dma_map and dma_unmap calls with the correct direction, or dma_alloc_coherent for long-lived buffers, and never touch a buffer while it is mapped to the device.
Register accesses that get cached or reordered. Symptom: a register read returns a stale status bit, or a sequence of register writes reaches the hardware in the wrong order. Cause: the region was mapped with normal cacheable attributes, or the driver used plain pointer writes instead of the proper accessors, so the compiler or the cache reordered or suppressed the accesses. Diagnose: the same read returns the same value forever even though hardware changed, or writes only take effect in bursts. Fix: map the region with device memory attributes and use readl and writel or ioread32 and iowrite32, which enforce single, ordered, non-speculative accesses.
Speculative access to device registers. Symptom: reading a FIFO or a clear-on-read status register at the wrong moment, or the system faults on an address the code never deliberately touched. Cause: the CPU speculates a load into device memory, or the compiler reorders it, because the attributes do not forbid speculation. Diagnose: the problem vanishes when the region is given stricter device attributes. Fix: use the strongest device type, Device-nGnRnE on Arm, for registers that are not safe to read speculatively, and keep all register traffic inside readl and writel.
The board boots the wrong image, or not at all. Symptom: after flashing new firmware the board still runs the old one, or a freshly assembled board stays silent with no console output. Cause: the boot pins select a different source than the one flashed, or an OTP fuse locks a source that no longer contains valid code. Diagnose: check the boot pins against the board's strap table, and read back the boot source the ROM actually chose, which many vendors print on the console as a line naming the device. Fix: set the straps to the intended source, and remember that a blown fuse is permanent, so only change fuses deliberately and on a known-good image.
The common thread is the memory map. When a driver misbehaves around memory, start from /proc/iomem to see how a region is claimed, then from dmesg to see how it was mapped, and finally from the device tree to see how the hardware was described. Memory ordering and cache bugs rarely print a helpful message, so the diagnosis is a process of comparing what the hardware did against what the attributes promised.
Try it
1 min readRead /proc/iomem and /proc/ioports on both your host and your board, and write out the physical map, naming the RAM range, the boot ROM or firmware range, and at least one peripheral window. Capture a PC UEFI boot log and an Arm ROM and TF-A boot log, and list the equivalent stages side by side as this lesson did. Finally, find the memory node in your board's device tree and confirm its reg value covers exactly the System RAM range shown in /proc/iomem. The practice pack checks those mappings and the coherency and memory-attribute reasoning against your own measurements.
Key takeaways
- The MMU turns each process's virtual addresses into physical addresses through multi-level page tables, cached in the TLB.
- Caches hide RAM latency, but dirty write-back lines and non-coherent DMA are the source of the hardest embedded bugs.
- An SoC memory map interleaves RAM, MMIO register regions and a small on-chip boot ROM, all visible in /proc/iomem.
- MMIO registers must be mapped with device memory attributes and touched with readl/writel, never as ordinary cached RAM.
- The boot ROM is immutable mask ROM that loads a first-stage loader from a source chosen by boot pins and OTP fuses.
- An x86 PC boots through firmware that reads an MBR or GPT partition to find a bootloader; ARM has no such standard.
- On Arm the full chain is ROM, TF-A, U-Boot, kernel, init, and Arm SystemReady is the push to give Arm a UEFI-class standard.
- The kernel lives in its own half of the address space, translated by a second page-table register, while each process gets its own map.
Reflect
- Explain to a colleague, in two sentences: explain why a virtual address is not the same as a physical address and which three problems the MMU solves.
- Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
- What surprised you when you ran the commands yourself?
On a real board
Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.
References and further reading
- Official docsLinux kernel documentation: Memory (arm64 virtual memory layout)
- Official docsLinux kernel documentation: Booting AArch64 Linux
- Man pageproc(5) manual page
- Official docsArm SystemReady certification program
- ArticleBootlin embedded Linux training slides (CC BY-SA 3.0)
Official documentation first. Links checked when the lesson was published.