Module 3 · Lesson 2 of 22

RISC-V for kernel programmers

The RISC-V you need to read xv6: registers and ABI, instruction formats, calling convention, privilege modes, CSRs, atomics, inline asm and GDB.

  • Beginner to advanced
  • 30 min read
  • 50 min lab
What you will need

A Linux machine or WSL2 with the xv6-riscv tree from the previous lesson built, QEMU (qemu-system-misc), gdb-multiarch and the riscv64 GNU toolchain; no physical board is required

By the end you can 7 objectives
  • Name the RV64GC extensions and the ABI role of each general register, and state which registers a callee must preserve.
  • Decode a 32-bit RISC-V instruction by hand into its R, I, S, B, U or J format fields.
  • Draw the stack frame produced by a typical prologue and epilogue and explain how a frame-pointer chain is walked.
  • Explain what M, S and U mode run in xv6 and what mret, sret, ecall and the main CSRs do at each privilege change.
  • Analyse how amoswap and lr/sc build a spinlock and a compare-and-swap, including the acquire and release ordering.
  • Apply gcc inline assembly and GDB disassembly to read entry.S, swtch.S, trampoline.S and riscv.h, and verify what the compiler generated.
  • Evaluate a faulty trap trace by reading scause, sepc and stval and deciding which privilege or CSR mistake caused it.
Read the lesson below; the video will appear here when published.
01

Why xv6 runs on RISC-V, and why you must read it

3 min read

The previous lesson gave you a map of xv6 and a booted system. This lesson gives you the vocabulary to read its lowest layer. Roughly a hundred lines of xv6 are not C at all: the first instructions that run after reset, the code that switches from one process to another, and the code that crosses between user mode and the kernel. A kernel programmer cannot skip them, because every crash, every race and every performance question eventually lands in those lines.

RISC-V is an open instruction set architecture (ISA), the contract between software and a processor: which instructions exist, what the registers are, and how privilege works. It began as a research project at the University of California, Berkeley, around 2010. The base integer instruction set and the privileged architecture were later frozen as ratified specifications (the main volumes in 2019), and since then a non-profit, RISC-V International, maintains them. Nobody pays a license to implement it, which is why companies large and small build RISC-V cores.

MIT chose RISC-V for the xv6 teaching kernel around 2019 for a practical reason: the ISA is small and regular. An older x86 version of xv6 spent pages of explanation on segmentation and legacy modes. The RISC-V version spends them on ideas. The same property helps embedded engineers. Microcontrollers such as the Espressif ESP32-C3 use 32-bit RISC-V cores, and boards such as the StarFive VisionFive 2 run full Linux on 64-bit RISC-V. Linux itself gained RISC-V support in version 4.15, released in early 2018.

xv6 runs on QEMU's virt machine with several harts, and every hart has its own copy of the registers you will meet below.

xv6 uses all three privilege modes that this lesson covers, which makes it unusual. A typical Linux system on RISC-V runs the kernel in supervisor mode and puts a small firmware layer in machine mode below it.

Privilege modes in xv6 and in Linuxxv6 on QEMULinux onRISC-VUser mode (U)shell, ls,user programsapplicationsSupervisor mode(S)whole xv6kernelwhole LinuxkernelMachine mode (M)entry.S andstart() onlyfirmware suchas OpenSBI
Which software runs in which RISC-V privilege mode, xv6 versus a typical Linux system
02

Registers and the RV64GC instruction set

3 min read

A register is a small, very fast storage slot inside the processor. RISC-V has 32 general-purpose integer registers named x0 to x31, and on a 64-bit machine each holds 64 bits. The name RV64GC says what you get: RV64 means a 64-bit base integer ISA (named I), and G is shorthand for a bundle of standard extensions: M for integer multiply and divide, A for atomic instructions, F and D for single and double precision floating point, plus two small extensions for control registers (Zicsr) and instruction-cache synchronisation (Zifencei). C adds compressed instructions, 16-bit encodings of the most common operations that make code smaller.

The kernel and the compiler agree on names for the registers. The RISC-V ABI gives each register a second name that tells you its job.

Register ABI name Role
x0 zero Always reads zero; writes are discarded
x1 ra Return address
x2 sp Stack pointer
x3 gp Global pointer (unused by xv6 kernel code)
x4 tp Thread pointer (xv6 stores the hart number here)
x5-x7 t0-t2 Temporaries, not preserved across calls
x8 s0 / fp Saved register, also the frame pointer
x9 s1 Saved register
x10-x11 a0-a1 Arguments and return values
x12-x17 a2-a7 More arguments (a7 also carries the system call number)
x18-x27 s2-s11 Saved registers
x28-x31 t3-t6 Temporaries

The program counter, pc, is separate: you cannot name it in an ordinary instruction, and only jumps, branches and traps change it. Register x0 is a clever trick. Because it always reads zero, addi a0, zero, 5 loads the constant 5, and a jump that discards its return address simply writes to x0. This is why the instruction set can stay small.

On RV64 the C type int is 32 bits and long and pointers are 64 bits (the LP64 data model). Instructions with a w suffix, such as addw, operate on the low 32 bits and sign-extend the result to 64 bits, which is how 32-bit int arithmetic is kept correct inside 64-bit registers.

Integer registers by ABI roleSpecialx0 zerox1 rax2 spx3 gpx4 tpTemporariest0-t2t3-t6Saveds0/fps1s2-s11Argumentsa0-a1a2-a7ra, t and a are caller-saved; sp and s are callee-saved
The 32 integer registers grouped by ABI role

Floating-point registers (f0 to f31) exist too, but the xv6 kernel does not use floating point, so you can ignore them while reading its source. A Linux kernel on RISC-V tracks a floating-point state field in sstatus so that it only saves those registers for processes that used them.

03

Instruction formats and the working set

6 min read

Every standard RISC-V instruction is 32 bits wide, and the low seven bits are the opcode. The remaining bits are laid out in one of six instruction formats: R, I, S, B, U and J. The designers kept the source register fields, rs1 and rs2, and the destination register field, rd, in the same bit positions in every format that has them. The processor can start reading the register file before it has finished decoding.

Format Used for Fields, high bit to low bit
R register-register arithmetic (add, sub) funct7, rs2, rs1, funct3, rd, opcode
I immediates, loads, jalr, CSR access imm[11:0], rs1, funct3, rd, opcode
S stores imm[11:5], rs2, rs1, funct3, imm[4:0], opcode
B conditional branches imm[12|10:5], rs2, rs1, funct3, imm[4:1|11], opcode
U lui, auipc imm[31:12], rd, opcode
J jal imm[20|10:1|11|19:12], rd, opcode

The funct3 and funct7 fields select the exact operation inside an opcode. Immediates are signed constants stored in the instruction itself. An I-type immediate is 12 bits, so it covers -2048 to 2047.

The six instruction formatsRfunct7rs2rs1funct3rdopcodeIimm[11:0]rs1funct3rdopcodeSimm[11:5]rs2rs1funct3imm[4:0]opcodeBimm[12,10:5]rs2rs1funct3imm[4:1,11]opcodeUimm[31:12]rdopcodeJimm[20,10:1,11,19:12]rdopcodebit 31 on the left; rs1, rs2 and rd never move
Bit layout of the six 32-bit instruction formats

The working set you will meet in xv6

You do not need all of the instruction set. These instructions cover almost everything in xv6:

  • addi rd, rs1, imm adds a signed 12-bit constant. It is also how mv (copy), li (small constants) and nop are built.
  • add, sub, and, or, xor, sll, srl, sra take two registers.
  • ld rd, off(rs1) and sd rs2, off(rs1) load and store 64 bits. lw, lwu, sw, lb, lbu, sb, lh, lhu and sh handle 32, 16 and 8 bits.
  • beq, bne, blt, bge, bltu and bgeu compare two registers and branch if the test is true. There is no flags register and no separate compare instruction.
  • jal rd, target jumps and writes the return address to rd. jalr rd, off(rs1) jumps to a register value plus an offset.
  • lui rd, imm puts a 20-bit value into bits 31 to 12 of rd. auipc rd, imm adds that shifted value to the current pc.
  • csrr, csrw, ecall, mret, sret and sfence.vma for the privileged work.

Many names you see in assembly are pseudoinstructions: Table of the common ones:

You write Real instruction
mv a0, a1 addi a0, a1, 0
li a0, 5 addi a0, zero, 5
j label jal zero, label
ret jalr zero, 0(ra)
call func auipc ra, hi then jalr ra, lo(ra)
beqz a0, label beq a0, zero, label
la a0, sym auipc a0, hi then addi a0, a0, lo (position independent)

Memory access uses exactly one addressing mode: a base register plus a signed 12-bit byte offset, as in ld a0, 16(sp). There is no indirect addressing through a memory location, no auto-increment and no scaled index. To walk an array, code adds to a pointer register with addi or add. Only loads and stores touch memory; arithmetic works on registers only. This is the load/store architecture that gives RISC its name.

Try decoding one by hand. The widget below holds addi sp, sp, -16, whose encoding is 0xff010113. Toggle bits and watch the fields change:

0xff010113 is the I-type instruction addi sp, sp, -16: opcode 0x13 (0010011), rd 2 (sp), funct3 0, rs1 2 (sp), imm 0xff0 which is -16 as a 12-bit signed number.

  • fields: [{"name": "opcode", "lsb": 0, "width": 7}, {"name": "rd", "lsb": 7, "width": 5}, {"name": "funct3", "lsb": 12, "width": 3}, {"name": "rs1", "lsb": 15, "width": 5}, {"name": "imm", "lsb": 20, "width": 12}]
  • value: 0xff010113
  • width: 32

Reading the result: opcode 0x13 is the arithmetic-immediate group, rd and rs1 are both 2 (sp), funct3 0 selects add, and the 12-bit immediate 0xff0 is -16 in two's complement. So the word means "sp = sp - 16". The same method decodes sd ra, 8(sp) as 0x00113423 (S-type: rs2 is ra, rs1 is sp) and ld ra, 8(sp) as 0x00813083 (I-type).

Going deeperWhy the B and J immediates look scrambled, and how li builds a 32-bit constant

Branch and jump targets are offsets from the pc and are always even, so bit 0 is never stored. The B and J formats scatter the remaining immediate bits so that most bits stay in the same instruction positions as in the S and I formats. This keeps the sign bit always at instruction bit 31, which lets hardware start sign-extending early. The price is that humans decode them with a table, and assemblers do it for you. Branches reach plus or minus 4 KiB, and jal reaches plus or minus 1 MiB; farther targets need an auipc and jalr pair.

To load a 32-bit constant, the assembler emits lui for the upper 20 bits and addi for the lower 12. Because addi sign-extends its immediate, a constant whose bit 11 is set needs the lui value rounded up by one. And on RV64, lui itself sign-extends bit 31 into the upper 32 bits, so lui a0, 0x80000 produces 0xffffffff80000000, not 0x80000000. A 64-bit address like the kernel base 0x80000000 therefore cannot be a plain lui: the assembler uses a longer sequence, and xv6's entry code uses la, which builds the address relative to pc.

04

The calling convention and stack frames

3 min read

When one function calls another, both were compiled separately, so they need rules about registers. The rules are the calling convention, part of the ABI. The caller puts the first eight integer arguments in a0 to a7 (more go on the stack), and the callee returns a result in a0, with a1 used for a second word of a 128-bit return value.

The important split is between two kinds of registers. These are ra, t0-t6 and a0-a7. These are s0-s11, and sp too. The stack pointer must also stay aligned to 16 bytes at function calls.

This split decides what a context switch saves. In xv6's swtch, you will see only ra, sp and s0-s11 stored. The caller of swtch is an ordinary C function, so by the convention it already saved everything else it cared about.

The stack frame

A stack frame is the block of stack memory a function uses for saved registers, local variables and spilled values. The stack grows toward lower addresses. A non-leaf function, one that calls others, must save ra because the call overwrites it.

asm
# typical shape of a non-leaf function (yours will differ)
func:
    addi sp, sp, -32      # prologue: allocate a 32-byte frame (multiple of 16)
    sd   ra, 24(sp)       # save the return address
    sd   s0, 16(sp)       # save the caller's frame pointer
    addi s0, sp, 32       # set the new frame pointer to the top of the frame
    # ... body, may call other functions ...
    ld   ra, 24(sp)       # epilogue: restore the return address
    ld   s0, 16(sp)       # restore the frame pointer
    addi sp, sp, 32       # free the frame
    ret                   # jalr zero, 0(ra)

With frame pointers enabled, s0 equals the stack pointer value at function entry. The saved ra then lives at fp-8 and the caller's saved fp at fp-16. That fixed layout is what makes a backtrace possible: read the return address at fp-8, move to the previous frame by loading fp-16, and repeat.

Step by stepOne call and return, register by register
  1. Caller sets arguments: it places arguments in a0-a7 and saves any t or a registers it still needs.

  2. jal ra, func: the call writes the address of the next instruction into ra and jumps to func.

  3. Prologue: func lowers sp by the frame size and stores ra and any s registers it will use.

  4. Body: func computes, possibly calling more functions, which repeat these steps with their own frames.

  5. Epilogue: func loads ra and the s registers back, raises sp by the same amount, and puts the result in a0.

  6. ret: jalr zero, 0(ra) jumps back to the saved address, and the caller continues with a0 holding the result.

One stack framehighCaller's frameSaved raat fp-8Saved caller fpat fp-16Locals and spillslowsp16-byte aligned, grows downhighCaller's frameSaved raat fp-8Saved caller fpat fp-16Locals and spillslowsp16-byte aligned, grows down
Layout of one stack frame with the frame pointer chain

A leaf function that calls nothing and needs few registers may skip the frame entirely. At optimisation level -O1, int add3(int a, int b, int c) returning a + b + c typically becomes two addw instructions and ret. You can confirm this on your machine with riscv64-linux-gnu-gcc -O1 -S. A frame pointer is not mandatory: compilers omit it at higher optimisation levels, and -fno-omit-frame-pointer brings it back at the cost of one register.

05

Privilege modes and control registers

5 min read

RISC-V defines privilege modes: levels of trust at which a hart executes. Three matter for xv6:

  • Machine mode (M) is the highest level and the one a hart starts in at reset. It can touch everything.
  • Supervisor mode (S) is where an operating system kernel normally runs. It controls virtual memory and handles most traps.
  • User mode (U) runs applications. It cannot execute privileged instructions or access privileged registers.

Compare this with ARM, which has exception levels EL0 to EL3. CPUs often have more than the two levels Linux uses; RISC-V's three are the same idea in a smaller package.

In xv6, only a few lines run in machine mode: _entry in entry.S and start() in start.c. They configure the hart and then execute mret to drop to supervisor mode, where the entire kernel runs. User programs run in user mode and enter the kernel through ecall. xv6 never returns to machine mode after boot, because it has no firmware services to call; QEMU is started with -bios none so no firmware sits underneath.

Control and status registers

The state that governs privilege, traps and memory lives in CSRs. You read one with csrr rd, csr, write one with csrw csr, rs, and atomically read and write with csrrw. They are not memory: no address, no load, no store.

CSR Purpose
sstatus Supervisor status: SIE (bit 1) enables interrupts, SPIE (bit 5) remembers it, SPP (bit 8) records the previous mode
stvec Address of the supervisor trap handler
sepc Pc saved at the trap; sret returns to it
scause Reason for the trap; top bit set means interrupt
stval Extra detail, usually the faulting address
sscratch A spare register for the trap handler to use before saving anything
satp Root of the page table and addressing mode
sie Which interrupt sources the supervisor will accept
mstatus, mepc, medeleg, mideleg The machine-mode equivalents; the last two delegate traps to supervisor mode

When a trap happens, the hardware does only a few things in order: it checks that interrupts are enabled if the trap is a device interrupt, clears SIE, copies pc to sepc, records the previous mode in SPP, sets scause (and stval where relevant), enters supervisor mode and sets pc to stvec. It does not save the other registers and does not change the page table. That minimalism is deliberate, and xv6's trap code in trampoline.S does the rest in software.

The scause codes you will see most are 8 for an ecall from user mode, 2 for an illegal instruction, and 12, 13 and 15 for page faults on instruction fetch, load and store. For an interrupt, the top bit of scause is set: supervisor timer is code 5 and supervisor external is code 9.

The ecall instruction deserves one note: sepc holds the address of the ecall itself, not the next instruction. If the kernel did nothing, sret would execute the ecall again. xv6's usertrap adds 4 to the saved pc before returning, which is why a system call returns to the instruction after it. ecall has no compressed form, so it is always 4 bytes.

Use the widget to follow how the mode changes. Fire ecall from user mode, then sret to go back:

Machine mode drops to supervisor mode with mret; supervisor mode returns to user mode with sret; user code reaches the kernel by executing ecall, which traps to supervisor mode, and a timer interrupt traps within supervisor mode.

  • states: [{"id": "machine", "label": "Machine mode"}, {"id": "supervisor", "label": "Supervisor mode"}, {"id": "user", "label": "User mode"}]
  • transitions: [{"from": "machine", "to": "supervisor", "event": "mret"}, {"from": "supervisor", "to": "user", "event": "sret"}, {"from": "user", "to": "supervisor", "event": "ecall"}, {"from": "supervisor", "to": "supervisor", "event": "timer interrupt"}]
Privilege transitionsMachine modeSupervisor modeUser modeTransitionsmretMachine modeSupervisor modesretSupervisor modeUser modeecall or interruptUser modeSupervisor modeecall to firmwareSupervisor modeMachine modeMachine modeSupervisor modeUser modeTransitionsmretMachinemodeSupervisormodesretSupervisormodeUser modeecall or interruptUser modeSupervisormodeecall to firmwareSupervisormodeMachinemode
How instructions and traps move a hart between M, S and U mode
06

Atomic instructions and the spinlock

4 min read

xv6 runs on several harts at once. If two of them increment the same memory word by loading, adding and storing, both may load the same old value and one update is lost. The fix is an atomic operation: The A extension provides two families.

Atomic memory operations (AMOs) perform a read-modify-write in one instruction: amoadd, amoand, amoor, amoxor, amomax, amomin and amoswap, each in .w (32-bit) and .d (64-bit) forms. For example, amoswap.w rd, rs2, (rs1) reads the word at the address in rs1 into rd, writes rs2 to that address, and the whole thing happens atomically.

Load-reserved and store-conditional build any other atomic update. lr.w rd, (rs1) loads a word and registers a reservation on that address. sc.w rd, rs2, (rs1) stores only if the reservation still holds; it writes 0 to rd on success and a nonzero value on failure. A compare-and-swap is a small loop:

asm
# cas_w(a0 = address, a1 = expected, a2 = new): returns 0 on success, 1 on failure
cas_w:
1:  lr.w  t0, (a0)        # load current value and reserve the address
    bne   t0, a1, 2f      # not the expected value: give up
    sc.w  t1, a2, (a0)    # try to store the new value
    bnez  t1, 1b          # reservation lost: retry
    li    a0, 0
    ret
2:  li    a0, 1
    ret

The spinlock built on one swap

xv6's spinlock needs less than compare-and-swap. A lock word is 0 when free and 1 when held. To acquire, atomically swap 1 into the word. If the old value that comes back was 0, this hart took the lock; if it was 1, someone else holds it, so loop and try again. That busy-wait loop is why it is called a spinlock.

Step by stepTwo harts race for one lock
  1. Both want the lock: hart 0 and hart 1 each execute amoswap, writing 1 to the lock word.

  2. Hardware orders them: the memory system performs one swap first; the other sees the result of the first.

  3. Winner sees 0: the first hart gets the old value 0 back and now owns the lock.

  4. Loser sees 1: the second hart gets 1 back, which means held, so it loops and swaps again.

  5. Release: the owner finishes its critical section and writes 0 to the word.

  6. Next winner: the spinning hart's next swap returns 0 and it enters the critical section.

c
// my own sketch of the idea, not xv6's code
int lock_word = 0;

void spin_acquire(void) {
    while (__atomic_exchange_n(&lock_word, 1, __ATOMIC_ACQUIRE) != 0)
        ;   // spin until the previous value was 0
}

void spin_release(void) {
    __atomic_store_n(&lock_word, 0, __ATOMIC_RELEASE);
}

xv6 itself uses the older gcc builtins __sync_lock_test_and_set and __sync_lock_release with __sync_synchronize() around them, and its book states that the swap boils down to a RISC-V amoswap instruction. The exact instruction the compiler emits depends on the gcc version, so check kernel/kernel.asm for your build rather than trusting a table.

Two harts, one lockHart 0Lock wordHart 1amoswap 1old value 0:acquiredamoswap 1old value 1: spinstore 0: releaseamoswap 1old value 0:acquired1. Hart 0 → Lock wordamoswap 12. Lock word → Hart 0old value 0: acquired3. Hart 1 → Lock wordamoswap 14. Lock word → Hart 1old value 1: spin5. Hart 0 → Lock wordstore 0: release6. Hart 1 → Lock wordamoswap 17. Lock word → Hart 1old value 0: acquired
Two harts racing for a lock with amoswap

Real RISC-V cores also have a weak memory model, called RVWMO: a hart can observe another hart's stores in a different order than they were written unless instructions say otherwise. That is why a lock must be more than an atomic swap, as the advanced section below shows.

Going deeperConstrained LR/SC loops and forward progress

The specification only guarantees that an lr/sc loop eventually succeeds if it is a constrained loop: at most 16 instructions, with the lr and sc in the same short sequence and no other memory accesses or taken branches that could break the reservation. Code outside those rules can livelock on some hardware, because a failed sc may keep failing. That is the reason atomic helpers are written in assembly or by the compiler's builtins, never by hand in the middle of unrelated code. AMOs have no such restriction, which is why simple spinlocks prefer amoswap.

07

Reading xv6's assembly and memory map

4 min read

The xv6 kernel directory has four files you can now read:

kernel/entry.S is the very first code. The hart starts in machine mode with no stack, so the C compiler's output cannot run yet. The assembly picks a stack for this hart: it loads the base of the stack0 array, reads mhartid with csrr, and computes sp = stack0 + (hartid + 1) * 4096, because the stack grows down from the top of that hart's slice. Then it calls start(). The pattern csrr a1, mhartid; addi a1, a1, 1; mul a0, a0, a1 is the whole trick.

kernel/start.c (C, with inline assembly) runs in M mode. It writes mstatus so that MPP names supervisor mode, writes mepc with the address of main, writes satp with 0 to leave paging off, delegates all exceptions and interrupts to supervisor mode with medeleg and mideleg, enables supervisor interrupts in sie, opens physical memory to S mode with the PMP registers, arms the timer, puts the hart number in tp, and finally executes mret.

kernel/swtch.S switches between kernel contexts. It is about thirty lines: a series of sd instructions saves ra, sp and s0 through s11 into the old context structure (the first argument, a0), then ld instructions load the same fourteen registers from the new context (a1), and ret jumps to the new ra. After the ret, the hart is running on a different stack, in a different function, as if returning from swtch.

kernel/trampoline.S holds the code that crosses between user and kernel space. On entry from a user trap, uservec writes a0 into sscratch to free a register, loads the fixed trapframe address, stores all 32 user registers there, loads the kernel stack and page table, and jumps to the C function usertrap. On the way out, userret restores everything and ends with sret. How the page tables make this safe is the topic of a later lesson; here only the registers matter.

Assembly-related files in xv6kernel/entry.Sstack per hart, call start()swtch.Ssave and load ra, sp, s0-s11trampoline.Suser and kernel trap crossingriscv.hCSR wrappers in inline asmmemlayout.hdevice and RAM addresseskernel/entry.Sstack per hart, callstart()swtch.Ssave and load ra, sp,s0-s11trampoline.Suser and kernel trapcrossingriscv.hCSR wrappers in inline asmmemlayout.hdevice and RAM addresses
The four assembly-related files and what each one does

The memory map

The file kernel/memlayout.h records the addresses of devices on the virt machine. You met the map in the previous lesson; here is what the kernel programs.

The kernel's device map from memlayout.h: CLINT at 0x02000000, PLIC at 0x0c000000, UART0 at 0x10000000, virtio disk at 0x10001000, and 128 MiB of RAM from 0x80000000.

  • regions: [{"name": "CLINT", "start": "0x02000000", "size": "0x10000", "detail": "Core-local interruptor: timer and software interrupts"}, {"name": "PLIC", "start": "0x0c000000", "size": "0x4000000", "detail": "Platform-level interrupt controller: routes device interrupts"}, {"name": "UART0", "start": "0x10000000", "size": "0x100", "detail": "Serial console, interrupt number 10"}, {"name": "VIRTIO0", "start": "0x10001000", "size": "0x1000", "detail": "Virtio disk, interrupt number 1"}, {"name": "RAM", "start": "0x80000000", "size": "0x08000000", "detail": "128 MiB, kernel loaded at 0x80000000"}]

The two interrupt devices are worth knowing now. The CLINT (core-local interruptor) provides a free-running timer, mtime, and per-hart timer compare and software-interrupt registers. The PLIC (platform-level interrupt controller) collects interrupts from devices such as the UART and the disk, lets software set priorities and enable bits per hart, and gives the hart a claim register to learn which device interrupted. Later sessions program both.

08

Inline assembly and disassembling with GDB

3 min read

C cannot read mhartid, so xv6 wraps single instructions in small functions in kernel/riscv.h using GCC's extended assembly. The general form is:

c
asm volatile("template" : outputs : inputs : clobbers);

The template is the instruction text, with %0, %1 standing for operands. The volatile keyword tells the compiler not to delete the statement or reorder it relative to other volatile statements. Output constraints begin with = (for example "=r" for a value written into some general register), input constraints name registers the compiler should load ("r"), and the clobber list names things the snippet damages, such as "memory" to forbid the compiler from caching memory values across it.

c
// reading a CSR into a C variable, in the style xv6 uses
static inline unsigned long read_sepc(void) {
    unsigned long x;
    asm volatile("csrr %0, sepc" : "=r" (x));
    return x;
}

// writing a CSR from a C value
static inline void write_satp(unsigned long x) {
    asm volatile("csrw satp, %0" : : "r" (x));
}

The real functions in riscv.h follow the same pattern with the names r_mhartid, w_satp and so on. intr_on and intr_off are even simpler: they read sstatus, set or clear the SIE bit in the C variable, and write it back. There is no magic, only a CSR and two bit operations.

Checking what the compiler generated

The compiler is not an oracle, and neither is your memory. Two tools let you verify.

The build already produces kernel/kernel.asm, a full objdump disassembly of the kernel with the C source interleaved. Search it for any function name. Or disassemble any object file yourself with riscv64-linux-gnu-objdump -d (the cross-tool prefix depends on your install).

The better tool is GDB attached to the running kernel. Start make qemu-gdb in one terminal and run gdb-multiarch kernel/kernel in a second, the same workflow as the previous lesson. Then:

gdb
(gdb) break swtch
(gdb) continue
(gdb) disassemble
(gdb) x/6i $pc
(gdb) stepi
(gdb) info registers ra sp
(gdb) p/x $sepc

disassemble lists the whole current function, x/6i $pc shows six instructions starting at the program counter, and stepi runs exactly one instruction (unlike step, which runs one C line). QEMU passes the CSRs to GDB, so $sepc and $scause normally work as convenience names; if your GDB build lacks them, info registers shows what it can see. On a hart with compressed instructions, the instruction addresses step by 2 or 4, and objdump prints the compressed ones with a c. prefix; a quick rule is that a 16-bit instruction has its two lowest bits not equal to binary 11.

GDB loop for checking codebreakcontinuex/6i $pcsee next instructionsstepioneinstructioninforegistersp/x $sepc↻ Repeat from step 1breakcontinuex/6i $pcsee next instructionsstepioneinstructioninforegistersp/x $sepc↻ Repeat from step 1
The loop for checking generated code and tracing a trap in GDB
Going deeperWhy the toolchain prefix and architecture string matter

Ubuntu offers gcc-riscv64-linux-gnu, whose tools are named riscv64-linux-gnu-*, and some setups use riscv64-unknown-elf-*. xv6's Makefile probes for whichever exists. You can ask a built binary what it targets with riscv64-linux-gnu-readelf -A kernel/kernel, which prints the architecture attribute string; this shows whether the compiler used the C extension and which others. An absolute rule: never mix a binary built for one architecture string with a disassembler's assumptions, or compressed instructions will be decoded as garbage.

09

Under the hood: CSR encoding, mret and sret, and acquire and release

5 min read

How a CSR instruction is encoded

csrr and csrw are pseudoinstructions for the underlying csrrs and csrrw. All CSR instructions are I-type with opcode 0x73 (SYSTEM): the 12-bit immediate field holds the CSR number, so there can be at most 4096 CSRs. For example, csrr a0, mhartid is csrrs a0, mhartid, zero and encodes as 0xf1402573, since mhartid is CSR 0xf14. And csrw satp, a0 is csrrw zero, satp, a0, encoded as 0x18051073, since satp is CSR 0x180.

The CSR number is structured. Bits 11:10 tell whether it is read-write (00, 01, 10) or read-only (11). Bits 9:8 give the lowest privilege that may access it: 00 user, 01 supervisor, 11 machine. So mhartid at 0xf14 has top bits 11 and is read-only and machine-level; sepc at 0x141 is read-write and supervisor-level. If code touches a CSR above its privilege, the hardware raises an illegal instruction exception. CSRs are the RISC-V trap-and-privilege mechanism because every privileged state sits behind this one narrow door, and the door checks the caller's mode.

CSR instruction and CSR numbercsrrwcsr[11:0]rs1funct3rd0x73CSR number11:10 RW or RO9:8 lowest mode7:0 indexmhartid 0xf14: read-only, machine level
Field layout of a CSR instruction and how the CSR number encodes access rules

Here are the numbers worth recognising when you read a disassembly: sstatus 0x100, sie 0x104, stvec 0x105, sscratch 0x140, sepc 0x141, scause 0x142, stval 0x143, satp 0x180, mstatus 0x300, medeleg 0x302, mideleg 0x303, mepc 0x341.

What mret and sret do

Both instructions return from a trap and perform three changes together. For sret: the pc is loaded from sepc; the privilege mode becomes whatever sstatus.SPP says (0 for user, 1 for supervisor); SIE is set from SPIE; SPIE is set to 1; and SPP is cleared to user. The mret instruction does the same with mepc, MPP, MIE and MPIE. This is why xv6 can enter its kernel main with a fake return: start.c writes MPP as supervisor and mepc as the address of main, then executes mret. The hart "returns" from a trap that never happened and lands in supervisor mode at main. The saved interrupt-enable state is the reason for the SIE/SPIE pair: a trap clears SIE so a handler is not interrupted, and the return restores the old value.

The only supervisor-mode view of paging is the satp register. Its top four bits select the mode (8 means Sv39, the three-level scheme xv6 uses), and the lower bits hold the physical page number of the root page table. xv6 starts with satp equal to 0, meaning no translation, and writes a real value when the kernel turns paging on. The structure of those tables is deferred to the memory lessons; for now, know that writing satp is one csrw and that code usually follows it with sfence.vma to flush stale cached translations.

Acquire and release: why the swap is the whole trick

Back in the spinlock, the atomic exchange gives mutual exclusion on its own, but modern cores reorder memory accesses. The critical section's stores and loads could be observed by another hart before the lock was taken, or after it was released. RISC-V gives AMOs two optional ordering bits: .aq (acquire) means no later memory operation may be observed before this one; .rl (release) means no earlier memory operation may be observed after it. An amoswap.w.aq that takes the lock therefore keeps the critical section's accesses below it, and a release store with .rl keeps them above it. xv6 reaches the same effect with the __sync_synchronize() call, which emits a full fence instruction.

So the whole spinlock is one swap and its ordering bits. Everything else, disabling interrupts to stop a handler from deadlocking against the lock holder, is built around that single instruction. Linux's own spinlocks on RISC-V rely on the same AMO and fence instructions, plus more elaborate queued variants.

Timer interrupts: CLINT and sstc

A hart's timer fires when a time counter reaches a compare value. Older xv6 programmed the CLINT's per-hart mtimecmp register in machine mode and used a small machine-mode handler to forward the event as a software interrupt to supervisor mode. The xv6 commit this course uses relies on the sstc extension, which lets supervisor mode write the stimecmp CSR directly, after machine mode enables it through menvcfg and mcounteren. Either way the kernel arms the next tick by writing the current time plus an interval; the tick that arrives is scause with the top bit set and code 5. The external-interrupt path always goes through the PLIC: the kernel reads the claim register, handles the named device, and writes the number back to complete it.

Going deeperHow Linux on RISC-V differs below the kernel

On a typical RISC-V board the first code is boot ROM, then a firmware stage, usually OpenSBI, which stays resident in machine mode, and then a boot loader such as U-Boot, and finally Linux. OpenSBI implements the RISC-V Supervisor Binary Interface: the kernel makes an ecall from supervisor mode with a function number in a7 and OpenSBI handles it in machine mode, for services such as timers on cores without sstc, inter-hart interrupts and system reset. Linux is entered in supervisor mode with the hart id in a0 and the physical address of the device tree in a1; Linux learns the hardware from the device tree, whereas xv6 hard-codes the virt map. xv6 skips all of this because QEMU's -bios none hands it the bare machine.

10

Common mistakes and how to diagnose them

3 min read

These are the failures that learners hit most often while reading and extending xv6 assembly. Each has a symptom, a cause, a way to confirm it, and a fix.

Diagram in production
Four common assembly mistakes with symptom, cause and first check

Stack not 16-byte aligned. Symptom: a function crashes on its first store, or only when called from your assembly. Cause: the ABI requires sp to be a multiple of 16 at calls, and a hand-written prologue used addi sp, sp, -8. Diagnose: print $sp in GDB at the crash and check the low four bits. Fix: allocate frames in multiples of 16.

Clobbering a callee-saved register. Symptom: a caller's loop variable or pointer changes after calling your assembly function, with no visible write. Cause: the function used s1 or s2 without saving and restoring it. Diagnose: single-step across the call with stepi and watch info registers s1. Fix: add the sd and ld pair in prologue and epilogue.

Forgetting that ecall leaves sepc on the ecall. Symptom: a system call runs forever in a loop, or the same call repeats. Cause: the handler returned without advancing the saved pc by 4. Diagnose: compare $sepc before and after the trap in GDB. Fix: the kernel increments the saved pc for syscalls, as xv6's usertrap does.

Treating a CSR like memory, or using the wrong privilege. Symptom: ld from a CSR number gives garbage, or user code that runs csrr takes an illegal instruction trap (scause 2). Cause: CSRs have no address, and csrr on a supervisor-level CSR is illegal in user mode. Diagnose: read scause and stval; scause 2 with the pc at your csrr instruction is the signature. Fix: use csrr and csrw in supervisor code only.

Reading the trap fingerprint. For any unexplained trap, record three values: scause, sepc and stval. scause tells the class (8 is ecall from user, 13 a load page fault, 15 a store page fault, and so on), sepc tells where it happened, and stval often tells the offending address. On Linux, a load page fault with stval near zero is almost always a null pointer dereference; xv6 is the exception, because it maps user text at address 0, so a null load reads code and only a null store faults. A store fault with a stval inside the stack's guard area is a stack overflow.

Other frequent slips are using li with a 64-bit constant and getting sign-extended surprises (see the lui note above), assuming int arithmetic leaves the upper 32 bits zero when it actually sign-extends, and expecting a branch to reach a far label: a bne to a label more than 4 KiB away fails to assemble with a relocation error, and the fix is an inverted short branch around a j.

11

Try it

1 min read

The lab for this lesson has three parts: write a small RISC-V assembly function, such as adding two integers or copying a string, and call it from a C user program in xv6; single-step a system call in GDB and record how a0 to a7, sepc, scause and stval change across the ecall; and hand-decode three instructions from the kernel disassembly and check them against the format tables above. Then work through the practice pack for this lesson, which has exercises, a quiz and interview questions. Next lesson, the reading skills you built here are applied to how xv6 boots, line by line.

Key takeaways

  • RV64GC is a 64-bit base integer ISA plus multiply, atomics, single and double float, and 16-bit compressed instructions.
  • Registers have ABI roles: ra, sp, a0-a7 for arguments, t0-t6 as temporaries, s0-s11 as callee-saved; x0 always reads zero.
  • Six instruction formats (R, I, S, B, U, J) keep register fields in fixed bit positions, so hardware decodes them cheaply.
  • A function prologue allocates the frame with addi sp and stores ra with sd; the epilogue reverses it and ends with ret.
  • Three privilege modes matter: machine firmware, supervisor kernel, user programs; mret and sret drop the privilege level.
  • CSRs such as sstatus, stvec, sepc, scause, stval, sscratch and satp are reached only with csrr and csrw, never with loads and stores.
  • amoswap makes xv6's spinlock work: one atomic exchange both tests and sets the lock, and its ordering bits make it acquire and release.
  • GDB's disassemble and x/i, plus kernel.asm, let you check the exact instructions behind any line of C.

Reflect

  1. Explain to a colleague, in two sentences: name the RV64GC extensions and the ABI role of each general register, and state which registers a callee must preserve.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close