Module 22 · Lesson 4 of 16

Serial ports (pyserial), SSH automation, sockets

Drive a board's UART with pyserial, automate SSH and SFTP with paramiko, and write TCP and UDP clients and servers with the socket module.

  • Beginner to advanced
  • 28 min read
  • 45 min lab
What you will need

Ubuntu 24.04 (WSL2, a VM, or a container) with Python 3.12 and the pyserial, paramiko, and requests packages; a USB-serial adapter and a Raspberry Pi for the full serial and SSH labs, or a socat loopback pair for the socket and serial labs without hardware

By the end you can 7 objectives
  • Explain what 8N1 means and how a UART frames each byte with start, data, parity, and stop bits.
  • Open, configure, read, and write a serial port with pyserial, using timeout and in_waiting correctly.
  • Encode outgoing text and decode incoming bytes safely, and read line-delimited logs and length-prefixed frames.
  • Run commands on a board with paramiko, capturing stdout, stderr, and the exit status, and verify host keys.
  • Transfer files with SFTP and run one command across many boards.
  • Write TCP and UDP clients and servers, and reassemble a partial-read message with framing.
  • Choose among raw sockets, requests, SSH, and serial for the job at hand.
Read the lesson below; the video will appear here when published.
01

Three doors into a board

3 min read

A board in a box has no keyboard and no screen. The Raspberry Pi on your bench, the router in a rack, and the test jig on a factory line all run Linux, and all of them are silent until you open a door into them.

The first door is the serial console. It runs over a serial port, two data wires plus a ground, and it works before the operating system finishes booting. When a board will not even start, the serial console is the only door that is open: the bootloader and the kernel both print their first messages there. You learned to watch those messages with a terminal program earlier in the program; this lesson replaces the human at the terminal with a script.

The second door is SSH. It runs over the network, so it needs the operating system and a network link to be up first. gives you a remote command line and file transfer, encrypted end to end.

The third door is the socket. A is how your own programs talk to the board, and how the finished product ships its data. This is not a door you walk through; it is the loading dock where your software moves cargo.

Three Python libraries line up with the three doors. pyserial opens the serial port. paramiko speaks SSH. The socket module in the standard library opens the socket door. You will meet each in turn, then learn when a higher-level library such as requests makes the socket door unnecessary.

The order matters on real hardware. A board boots, and you watch its boot log on the serial console. Once the network is up, you use SSH to push files and run provisioning commands. Finally, the running product talks over a socket you designed. Board bring-up almost always happens in exactly that order, and a script that automates all three is the tool that lets one engineer provision a hundred boards instead of sitting at each one.

Three doors into a boardSerialconsoleSSHRaw socketReaches theboardbefore the OSbootsafter OS andnetwork areupwhen yourprogram runsPython toolpyserialparamikosocket moduleEncryptednoyesonly if youadd TLSBest forboot logs,bring-upprovisioning,file transferthe runtimeprotocol
The three ways to reach a board, compared by when they work and what each is best for.
02

The UART and what 8N1 means

5 min read

Before you open a serial port in Python, understand the wire, because the most common serial bug is a simple mismatch of settings.

A is asynchronous. That word means there is no clock wire. Nothing on the wire says "now is the middle of a bit". Instead, both ends agree on a speed and a frame format in advance, and the transmitter encodes each byte inside start and stop bits so the receiver can find the byte edges by itself.

The is that agreed speed. Common values are 9600, 19200, 38400, 57600, and 115200; 115200 is the usual Linux board console speed, and 921600 and faster appear on high-throughput links. Both ends must match exactly. If they do not, the receiver samples each bit at the wrong moment and prints garbage, a stream of wrong characters. When you see mojibake on a console, the baud rate is the first thing to suspect.

Here is how one byte travels, with the line idle at the high level, which engineers call mark. A start bit drops the line low, called space, for one bit time. Then the data bits follow, least significant bit first. An optional parity bit may follow. Then one or more stop bits hold the line high for at least one bit time, and the line returns to idle. The receiver watches for the falling edge of the start bit and then samples each following bit at its centre.

The three settings in the shorthand 8N1 are data bits, parity, and stop bits. 8 data bits carry the whole byte, which is what you want for binary data and for UTF-8 text. is N for none, E for even, or O for odd. Even parity sets the parity bit so the total count of 1s in the data plus parity is even. One stop bit is standard; two stop bits were for old mechanical equipment that needed a longer rest between bytes. The configuration 8N1, meaning 8 data bits, no parity, one stop bit, is the default everywhere, from a board console to a GPS module.

The board's UART appears in Linux as a device node under /dev, and the name tells you the driver. /dev/ttyS0 through /dev/ttyS3 are the classic 8250/16550-compatible ports, the old PC COM ports. /dev/ttyUSB0 is a USB-serial adapter, a chip such as an FTDI, CP210x, CH340, or PL2303 that gives a laptop a serial port over USB. /dev/ttyAMA0 is an ARM AMBA PL011 UART, the kind built into Raspberry Pi systems on a chip. /dev/ttyACM0 is a USB CDC-ACM device, a modem-style port. Your board may name it differently, so confirm the node with dmesg or with pyserial's own device listing, which you will meet shortly.

The two data wires are crossed: the transmitter's TX goes to the receiver's RX, and the other way around, with a common ground. Crossing the pair wrong, or forgetting the ground, is the second classic wiring mistake, right after the baud mismatch.

Step by stepHow one byte crosses a UART wire
  1. Line idles high: with nothing to send, the wire rests at the high level, called mark.

  2. Start bit: the transmitter drops the line low for exactly one bit time, telling the receiver a byte is coming.

  3. Data bits: the transmitter sends the eight data bits least significant first, one per bit time.

  4. Optional parity bit: if parity is configured, one parity bit follows the data bits.

  5. Stop bit: the transmitter returns the line high for at least one bit time.

  6. Idle again: the line stays high until the next start bit, or the next byte follows.

The 8N1 frame on the wireidleline highstart bitline low8 data bitsLSB firstparitynone (8N1)stop bitline highidleor nextstart bitidleline highstart bitline low8 data bitsLSB firstparitynone (8N1)stop bitline highidleor nextstart bit
The 8N1 frame: an idle-high line, a low start bit, eight data bits, no parity, and one stop bit.
03

Opening a port with pyserial

4 min read

pyserial is the standard library for serial ports, a thin Python layer over the operating system's own serial support. Install it once with pip install pyserial. Then opening a port is one constructor call:

python
import serial

with serial.Serial(
    "/dev/ttyUSB0",
    baudrate=115200,
    bytesize=serial.EIGHTBITS,
    parity=serial.PARITY_NONE,
    stopbits=serial.STOPBITS_ONE,
    timeout=1,
    write_timeout=1,
) as port:
    port.write(b"uname -a\n")
    line = port.readline()
    print(line.decode("utf-8", errors="replace").rstrip())

Every argument maps to something you met in the last section. port names the device node. baudrate is the speed, and both ends must agree. bytesize is serial.EIGHTBITS by default, with FIVEBITS through SEVENBITS also available for old equipment. parity and stopbits default to PARITY_NONE and STOPBITS_ONE. The serial.Serial object is a context manager, so the with statement closes the port even if the block raises, exactly like opening a file.

The two timeout arguments deserve the most attention, because they behave differently from sockets and from each other. timeout limits how long a read waits, and it accepts three kinds of value: None blocks forever, 0 means do not wait at all, and a positive number waits that many seconds. The key fact is that a timed-out read is not an error. read(n) returns the bytes it managed to get, which may be fewer than n, or empty. It never raises. write_timeout limits how long a write may take, and a write that exceeds it raises serial.SerialTimeoutException. The split exists because a stuck write is usually a real fault, while a quiet read is usually just a board with nothing to say yet.

So the correct reading pattern is to ask for more than you expect and accept less. read(n) blocks until n bytes arrive or timeout passes. readline() reads until a newline or timeout. read(1) reads one byte. And in_waiting tells you how many bytes are already sitting in the input buffer, so read(port.in_waiting) returns everything waiting without blocking at all:

python
import serial, time

with serial.Serial("/dev/ttyUSB0", 115200, timeout=0.1) as port:
    while True:
        waiting = port.in_waiting
        if waiting:
            data = port.read(waiting)
            print(data.decode("utf-8", errors="replace"), end="")
        else:
            time.sleep(0.01)

That loop is a serial console logger in miniature. It polls in_waiting, reads only what is there, and never blocks on a silent line. It is not the most efficient design; a full logger reads in a dedicated thread and handles structured binary messages, which the next section and the lab build.

Before any read loop can work you must be allowed to open the node. On Ubuntu and Raspberry Pi OS the serial nodes belong to the dialout group, so add yourself with sudo usermod -aG dialout $USER and log in again, or run with sudo. On a fresh board the node may also appear under a different name; serial.tools.list_ports, coming next, is how to find it.

Going deeperThe tty layer underneath pyserial

pyserial is not doing the electrical work itself. Underneath sits the Linux tty layer, the same subsystem that handles terminal programs. pyserial sets the port to raw mode and programs the driver through termios ioctls: the baud rate, character size, parity, and stop bits you passed become values in a termios structure. The UART driver underneath is usually an 8250/16550-compatible part, and that chip has a small receive FIFO, 16 bytes in the classic 16550. When your program is slow to read, the FIFO fills, the driver's software buffer fills, and incoming bytes are dropped, an event called an overrun. You can see per-port error counts on many systems with cat /proc/tty/driver/serial, and the hardware handshake covered next exists to prevent exactly this loss.

The serial stackYour scriptserial.Serialpyserialraw mode, termios ioctlstty driver16550-compatible UARTUART hardwareTX, RX, RTS, CTSwrites go down, bytes come upYour scriptserial.Serialpyserialraw mode, termios ioctlstty driver16550-compatible UARTUART hardwareTX, RX, RTS, CTSwrites go down, bytes come up
The layers a serial write passes through: your script, pyserial, the tty driver, and the UART hardware.
04

Bytes on the wire: text, frames, and flow control

3 min read

The previous lesson drilled one rule, and the serial port obeys it too: on the wire there are only bytes, never text. serial.Serial.write accepts bytes and returns the count written:

python
port.write(b"help\n")                    # fine, bytes
port.write("help\n")                     # TypeError: write() argument must be bytes
port.write("help\n".encode("utf-8"))     # fine, text encoded first

Outgoing text must be encoded to bytes, and UTF-8 is the encoding modern systems use. Incoming bytes must be decoded, and you should pass errors="replace" so one corrupted byte prints a replacement character instead of crashing your logger with a UnicodeDecodeError. Data that is not text, such as a firmware image, a GPS binary sentence, or a protocol frame, stays as bytes and is never decoded.

How you read depends on what the board sends. Logs and command output are line-delimited: the board prints one line, ending in a newline. readline() and read_until(b"\n") handle that. Protocols are usually framed. is the difference between a stream of characters and a sequence of messages. The two common rules are a delimiter, such as a newline or a special byte, and a length prefix, where each message begins with a fixed-size field stating how many bytes follow:

python
import struct

def read_frame(port):
    header = port.read(2)                # 2-byte little-endian length
    if len(header) < 2:
        return b""                       # timeout, no full frame
    (n,) = struct.unpack("<H", header)
    payload = bytearray()
    while len(payload) < n:
        chunk = port.read(n - len(payload))
        if not chunk:
            break                        # timeout mid-frame, partial data
        payload += chunk
    return bytes(payload)

The loop matters because read(n) may return fewer than n bytes, the same partial-read behaviour you will meet again with sockets. It keeps reading until the frame is whole, and a mid-frame timeout is the moment you must decide to discard or retry. The struct module that unpacks the length is the subject of the next lesson; here you only need to know that "<H" reads two bytes as a little-endian unsigned number.

Before starting a fresh exchange, clear the pipe. A board that booted while your script was not listening has already printed a backlog. flush() waits for the write buffer to drain, and reset_input_buffer() throws away unread incoming data so your next read starts at a known point. The older names flushInput() and flushOutput() do the same two jobs.

Two control lines round out the port. DTR and RTS are extra wires, separate from TX and RX, that the host can raise and lower with port.dtr = True and port.rts = True. Be careful with DTR on developer boards: many of them, including Arduino and ESP32 designs, wire DTR to the reset pin, so the act of opening the port toggles it and resets the board. If that surprises you, set port.dtr = False before opening, or accept the reset and wait for the boot output.

is how a host keeps up with a fast data source. The hardware kind uses RTS and CTS: the receiver lowers RTS when its buffer is nearly full, and the sender sees CTS drop and stops, so the overrun from the last section never happens. Pass rtscts=True to enable it. The software kind sends the special bytes XOFF (0x13) to pause and XON (0x11) to resume, enabled with xonxoff=True; it cannot be used inside binary data, because those byte values might appear in the payload. If your data is latency-sensitive, read continuously rather than polling slowly, and enable hardware flow control so nothing is dropped while you are busy.

Finally, find the right device before you open anything. serial.tools.list_ports.comports() returns one object per detected port, each carrying .device (the node), .description (the chip and its name), and for USB parts .vid, .pid, and .serial_number. Use it to pick the one adapter you own out of several:

python
from serial.tools import list_ports

for p in list_ports.comports():
    print(p.device, "-", p.description, "-", p.serial_number)
Text to bytes and backstrthe text tosendencodeUTF-8 tobyteswriteinto the TXbufferUART wirestart,data, stopbitsreadbytes comebackdecodeUTF-8, errors=replacestrthe text tosendencodeUTF-8 tobyteswriteinto the TXbufferUART wirestart,data, stopbitsreadbytes comebackdecodeUTF-8, errors=replace
The path of a message: text is encoded to bytes, written to the UART, and decoded back on read.
05

Automating SSH with paramiko

4 min read

The serial console is for bring-up. Once the board has a network, the day-to-day door is SSH, and paramiko is the Python library that speaks the SSH protocol. Install it with pip install paramiko, then run a command on a remote board:

python
import paramiko

client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.RejectPolicy())
client.connect("labpi", port=22, username="learner",
               key_filename="/home/learner/.ssh/id_ed25519")
stdin, stdout, stderr = client.exec_command("uname -a")
out = stdout.read()          # bytes
err = stderr.read()          # bytes
status = stdout.channel.recv_exit_status()
client.close()
print(status, out.decode().strip())

Four lines do the security work, so understand each. load_system_host_keys reads the known_hosts file, the list of server keys you have seen before. set_missing_host_key_policy decides what happens when a key is not in that list. connect establishes the connection and authenticates as learner with a private key file. The three streams from exec_command are file-like objects in bytes mode: stdin lets you write to the command, stdout carries its output, and stderr carries its diagnostics.

The order at the end is not optional. Read stdout and stderr fully before calling recv_exit_status. The remote command streams output over a channel with a limited buffer, and if you ask for the exit status first, the remaining output can fill the channel and the whole call hangs. Drain the streams, then collect the status. A status of 0 means success; anything else is the command's exit code, exactly as you learned with subprocess.

Authentication has two forms. A password is the obvious one, passed as password="...", and it is the form to avoid in automation: the password sits in your script or a config file in plain text, and a human often has to type it. A key pair is the better form. You generate it once with ssh-keygen, the private key stays on your machine, the public key goes into the board's authorized_keys file, and connect proves who you are with key_filename= or by loading the key yourself:

python
key = paramiko.Ed25519Key.from_private_key_file("/home/learner/.ssh/id_ed25519")
client.connect("labpi", username="learner", pkey=key)

Keep the private key file readable only by you, with chmod 600, the same rule OpenSSH enforces.

The other half of SSH security is the . On the first connection to a new host, no recorded key exists, and the policy decides what happens. RejectPolicy, the default, refuses the connection, which is the safe choice: you seed the known_hosts file with the real key first, for example with:

bash
ssh-keyscan labpi >> ~/.ssh/known_hosts

Verify that key out of band before trusting it. AutoAddPolicy accepts any unknown key and records it, convenient for a throwaway test but dangerous for unattended automation, because it teaches your script to accept whatever host answers, including an attacker in the middle. WarningPolicy accepts but logs. For scripts that must run without you, seed the known_hosts file and keep RejectPolicy.

Going deeperSSH-2 authentication is a signature, not an encryption

A persistent myth says the server encrypts a challenge with your public key and you decrypt it with the private key. SSH-2 does not work that way, and it cannot: Ed25519 keys cannot encrypt at all. What actually happens, per RFC 4252, is that after the key exchange establishes a shared session key, the client signs data with its private key and the server verifies the signature with the stored public key. A valid signature proves possession of the private key, without the private key ever leaving your machine. The host key check works the same way: the server proves its identity by signing with its host key, which is why a mismatched host key is a serious warning, not a cosmetic one.

One SSH commandScriptBoard (SSH server)ShellTCP connect :22host key + keyexchangesign with privatekeyauth OKexec_command requestrun commandstdout, stderroutput + exit status1. Script → Board (SSHserver)TCP connect :222. Board (SSH server) →Scripthost key + key exchange3. Script → Board (SSHserver)sign with private key4. Board (SSH server) →Scriptauth OK5. Script → Board (SSHserver)exec_command request6. Board (SSH server) → Shellrun command7. Shell → Board (SSH server)stdout, stderr8. Board (SSH server) →Scriptoutput + exit status
An SSH session: connect, key exchange and host-key check, signature authentication, then command output and exit status.
06

SFTP and many boards at once

3 min read

Underneath the SSHClient sits one encrypted connection, which paramiko calls a transport. That single connection multiplexes many channels, and each exec_command or file transfer is a channel riding on top. Seeing the transport as the real object is what unlocks two advanced tools: interactive sessions and port forwarding.

exec_command opens a session channel and asks the remote to run one command, with no terminal attached. Some programs need a terminal anyway: an interactive shell, a menu, a tool that prompts. For those, invoke_shell opens a session channel with a pseudo-terminal, a PTY, attached:

python
chan = client.invoke_shell()
chan.send(b"uname -a\n")
time.sleep(0.5)
print(chan.recv(4096).decode(errors="replace"))

invoke_shell is for driving an interactive program step by step, reading whatever it prints after each keystroke. It is slow and fragile compared with exec_command, so use it only when a program genuinely refuses to run without a terminal. For a one-shot command that merely wants a TTY, exec_command(..., get_pty=True) is the middle ground.

A channel can also be a tunnel. If the board's network has a service reachable only from the board itself, a direct-tcpip channel forwards a TCP connection through the SSH connection to it:

python
transport = client.get_transport()
chan = transport.open_channel("direct-tcpip", ("192.168.1.10", 80), ("", 0))
chan.send(b"GET / HTTP/1.0\r\n\r\n")
print(chan.recv(1024).decode(errors="replace"))

The first tuple is the destination on the remote side, the second is the local side. One open_channel call reaches a web server on the board's network without opening any port on your machine or the board.

File transfer is the third lane. is a subsystem of SSH, not FTP. It opens with open_sftp and copies files with put and get:

python
sftp = client.open_sftp()
sftp.put("/home/learner/provision.sh", "/home/learner/provision.sh")
sftp.chmod("/home/learner/provision.sh", 0o755)
sftp.close()

stdin, stdout, stderr = client.exec_command("bash /home/learner/provision.sh")
out = stdout.read(); err = stderr.read()
status = stdout.channel.recv_exit_status()

That pattern, push a script and run it, is the provisioning lab in miniature: no interactive copy, no passwords, and the result comes back with an exit code you can branch on. Run it across a fleet by looping, one client per board:

python
boards = ["labpi-01", "labpi-02", "labpi-03"]
results = {}
for host in boards:
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.RejectPolicy())
    client.connect(host, username="deploy",
                   key_filename="/home/learner/.ssh/id_ed25519")
    _, out, _ = client.exec_command("fw_printenv bootcount")
    results[host] = out.read().decode().strip()
    client.close()

Sequential is fine for a few boards, and fw_printenv is the U-Boot tool that reads a value from the board's boot environment. For a hundred boards, run the body inside a concurrent.futures.ThreadPoolExecutor, because the network round-trip is the slow part and threads overlap it well; asyncio does the same job without threads and arrives in session S04B. Give unattended scripts their own user: a least-privilege account such as deploy, with key-only login and permission for exactly the files and commands it needs, not a shared root key.

Channels on one SSH transportSSH transportone encrypted connectionsession: execexec_commandsession: shellinvoke_shell, PTYdirect-tcpipport forwardsftp subsystemget, putSSH transportone encrypted connectionsession: execexec_commandsession: shellinvoke_shell, PTYdirect-tcpipport forwardsftp subsystemget, put
One SSH transport multiplexing channels: an exec session, an interactive shell, a port forward, and the SFTP subsystem.
07

Raw sockets: TCP and UDP

3 min read

A connection is the workhorse of device networking. The socket module is in the standard library, and a TCP client is five lines:

python
import socket

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect(("labpc", 5000))
    s.sendall(b"hello\n")
    data = s.recv(1024)
print(data)

AF_INET is IPv4, and SOCK_STREAM is TCP. The tuple ("labpc", 5000) is the remote address and port; the hostname is resolved for you. sendall sends all the bytes, looping internally if the kernel accepts them in pieces; the lower-level send does not loop and may send fewer, which is why sendall is the safe default. recv(1024) returns up to 1024 bytes. The socket object is a context manager, so the with statement closes it even when the block raises, the same guarantee you rely on for files.

The server side listens instead of connecting:

python
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as srv:
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(("0.0.0.0", 5000))
    srv.listen(5)
    conn, addr = srv.accept()
    with conn:
        data = conn.recv(1024)
        conn.sendall(data)          # echo it back

bind fixes the local address and port, listen puts the socket into listening mode with a backlog of pending connections, and accept blocks until a client connects, returning a new socket for that one connection plus the client's address. SO_REUSEADDR lets you restart the server immediately instead of waiting for the old socket to time out, a constant annoyance during development. The echo loop above answers one connection and exits; the lab extends it to many.

is the other half of the module. It needs no connect and no listen:

python
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
    s.sendto(b"ping", ("labpc", 5000))
    data, addr = s.recvfrom(1024)

A UDP server binds, then loops over recvfrom, which returns the datagram and the sender's address so you can reply with sendto. Because there is no connection, one socket serves many senders, and a datagram arrives whole or not at all, its boundary intact. The trade is that UDP does not guarantee delivery, order, or de-duplication, so anything important needs its own retries and sequence numbers on top.

Step by stepHow a TCP connection is made and ended
  1. SYN: the client sends a synchronize packet carrying a random sequence number.

  2. SYN-ACK: the server replies, acknowledging the client's number and sending its own.

  3. ACK: the client acknowledges the server, and the connection is established.

  4. Data: both sides exchange bytes over the established connection.

  5. FIN: one side sends finish and the other acknowledges, closing that direction.

  6. TIME_WAIT: the closer waits briefly so late packets cannot confuse a future connection.

TCP versus UDPTCP(SOCK_STREAM)UDP(SOCK_DGRAM)Connectionyes,three-wayhandshakenoneDeliveryguaranteed,in ordernotguaranteedMessageboundariesnone, a bytestreampreserved perdatagramPython callsconnect,sendall, recvsendto,recvfrom
TCP and UDP compared on connection, delivery, boundaries, and the Python calls each uses.
08

Timeouts and the framing problem

3 min read

Sockets block by default. connect, accept, and recv all wait until they can finish, which is why a script can hang silently for minutes against a dead host. The first defence is a timeout. settimeout(seconds) makes every following blocking call give up and raise TimeoutError after that long, and settimeout(None) restores blocking. The second defence, for a socket that must never block, is setblocking(False), after which a call that cannot proceed raises BlockingIOError immediately, so you can go do something else and try again later.

Even with a timeout, TCP has a trap that catches everyone once: recv(1024) returns up to 1024 bytes, not exactly 1024, and not one message. TCP is a stream. The kernel makes no promise about how your sendall calls map to your recv calls. Two sendall calls can arrive in a single recv; one message can arrive split across three recv calls. A client that sends b"ping" and b"pong" back to back can hand the server "pi", then "ngpo", then "ng". The boundaries you sent are gone.

The fix is framing, the rule you met on the serial port, applied here with the same discipline. The most robust rule is a length prefix, a fixed-size header stating the payload length, followed by that many payload bytes:

python
import struct

def recv_exact(conn, n):
    buf = bytearray()
    while len(buf) < n:
        chunk = conn.recv(n - len(buf))
        if not chunk:
            raise ConnectionError("peer closed mid-message")
        buf += chunk
    return bytes(buf)

def recv_message(conn):
    header = recv_exact(conn, 4)
    (n,) = struct.unpack("!I", header)   # 4-byte big-endian length
    return recv_exact(conn, n)

The pattern is in three steps, and each is a distinct idea.

1read exactly N bytes

recv may return fewer than you asked, so recv_exact loops until it has all n bytes, treating an empty return as the peer closing the connection. This single helper removes the partial-read bug from the rest of your code.

2separate length from payload

struct.unpack("!I", header) reads four bytes as a big-endian unsigned integer, the network byte order. Big-endian is the convention on the wire, so both ends must agree on it, and "!" pins it regardless of the machine you run on. struct is the next lesson's topic; here it is only a length codec.

3frame, then dispatch

The receiver reads a header, then exactly the declared payload, and only then hands a complete message to the application. Delimiter framing, splitting on a newline or a special byte, is the simpler alternative when the payload can never contain the delimiter, and makefile() turns a socket into a file-like object so readline() works. For variable binary payloads, prefer the length prefix.

Non-blocking sockets and select are the path from one connection to many. A single-threaded server can hold many sockets, run setblocking(False) on each, and use selectors to be told which ones are ready, instead of blocking on any one of them. That idea, waiting for many things at once instead of many threads, is exactly what asyncio builds on in session S04B, so this lesson leaves you one step away from it.

Going deeperWhy tiny sends are slow, and why TIME_WAIT exists

Three kernel facts explain most socket weirdness. First, each side has a fixed send buffer and receive buffer, and sendall blocks when the send buffer is full, not when the peer has read the data; the data has left your program, but not necessarily the machine. Second, two old optimisations, Nagle's algorithm and delayed acknowledgements, deliberately hold small segments briefly to avoid flooding the network with tiny packets, which can add surprising latency to a control loop that sends one small message at a time; setting TCP_NODELAY disables the sender's half when latency matters more than throughput. Third, the TIME_WAIT state exists so a delayed old packet from a closed connection cannot be mistaken for data on a new one, and it is why a restarted server hits address already in use without SO_REUSEADDR.

Reassembling one messagerecvup to nbytesappendto thebuffer4-byteheader?big-endianlengthreadpayloadexactly nbytesdispatchonecompletemessagerepeatfrom recv↻ Repeat from step 1recvup to nbytesappendto thebuffer4-byteheader?big-endianlengthreadpayloadexactly nbytesdispatchonecompletemessagerepeatfrom recv↻ Repeat from step 1
The reassembly loop: recv, append to a buffer, read the length header, collect the payload, then dispatch one whole message.
09

Addresses, hostnames, and when HTTP is enough

2 min read

So far every example passed a hostname and let the library resolve it. Resolution is worth one explicit look, because device networks are where it breaks. socket.getaddrinfo is the modern resolution call:

python
for family, socktype, proto, canonname, sockaddr in socket.getaddrinfo("labpc", 22):
    print(family, socktype, sockaddr)

It returns every way to reach labpc on port 22, one tuple per candidate. family is AF_INET for IPv4 or AF_INET6 for IPv6, and sockaddr is ("192.168.1.10", 22) for the first and ("fe80::1", 22, 0, 0) for the second. You loop over the list and try each candidate until one connects, which is the correct way to support both address families. The older gethostbyname returns only the first IPv4 address and cannot express IPv6 or a choice, so prefer getaddrinfo.

A hostname is a name, an IP address is a location, and the distinction matters when a board moves. A script that hardcodes 192.168.1.10 breaks when the DHCP lease (the address the network assigned it) changes; a script that uses a name keeps working as long as DNS or mDNS resolves it. On small device networks without DNS, mDNS names ending in .local, the Avahi service, or an /etc/hosts entry are the usual substitutes.

The practical question at the end of this lesson is whether you need raw sockets at all. For an ordinary HTTP request, you do not. A raw client for one GET is dozens of fiddly lines handling the request line, headers, and the read loop, while requests does it in one:

python
import requests
r = requests.get("http://labpc:8000/status", timeout=5)
print(r.status_code, r.json())

requests handles DNS, connection reuse, redirects, headers, TLS (the encryption that turns HTTP into HTTPS), and JSON decoding, and its timeout parameter gives you the safety you had to build by hand with sockets. httpx is the newer sibling, with the same API plus async support and HTTP/2, and it is the library the REST work in session S04B builds on. The rule is simple: if the board speaks HTTP, use requests or httpx; if it speaks a protocol you designed, or you need exact control over bytes and framing, use raw sockets. The framing skills from the last section are not wasted, because every protocol you design will need them.

Raw socket versus requestsRaw socketrequests /httpxOne HTTP GETdozens oflinesone lineHandles for younothingDNS, TLS,redirects,JSONUse whenyour ownprotocol,exact bytesthe boardspeaks HTTP
A raw socket versus requests for one HTTP GET: what each handles for you and when to reach for which.
10

Secure defaults and common mistakes

3 min read

The mistakes in this lesson's territory are few and well known. Here they are as symptom, cause, diagnosis, and fix.

Garbage on the console. Cause: a baud rate or frame mismatch, usually the port at 115200 while the board speaks 9600, or a wiring swap. Diagnose: the wrong characters are consistent, not random corruption. Fix: try the common rates in turn, confirm 8N1, and check that TX crosses to RX with a shared ground.

Permission denied on the device node. Cause: your user is not in the dialout group. Diagnose: ls -l /dev/ttyUSB0 shows the owning group. Fix: sudo usermod -aG dialout $USER and log in again, or add a udev rule for your specific adapter.

The board resets the moment the port opens. Cause: a DTR line wired to the board's reset pin, common on Arduino and ESP32 boards. Diagnose: the board reboots exactly on port open. Fix: set port.dtr = False before opening, or wait for the boot output before talking.

The SSH script hangs after exec_command. Cause: reading the exit status before draining stdout and stderr. Diagnose: the process sits in recv_exit_status forever. Fix: read both streams first, then collect the status.

A host key warning every run, or an accepted attacker. Cause: AutoAddPolicy, which records whatever host key arrives first. Diagnose: the script never fails a host check. Fix: seed known_hosts with the real key and use RejectPolicy; this is the single most important secure default in unattended automation.

A password or key in the repository. Cause: convenience. Diagnose: a search of the repository history finds it. Fix: key-only authentication with a least-privilege user, the private key readable only by its owner, and no secrets in logs, the rule from the logging lesson.

Address already in use on server restart. Cause: a socket in TIME_WAIT. Diagnose: it fails only on the second immediate start. Fix: set SO_REUSEADDR before bind.

Half a message arrives. Cause: treating recv as if it returned exactly one message. Diagnose: the first recv is short, and the rest arrives next time. Fix: frame your messages with a length prefix or delimiter and loop until complete.

A script blocks forever against a dead board. Cause: no timeout on connect, recv, or the serial read. Diagnose: the process is stuck in a network call. Fix: set a timeout everywhere, and treat a timeout as a result to handle, not an uncaught crash.

These nine failures share three secure defaults that you should set before the first board is attached: verify host keys strictly, authenticate with keys only, and give automation its own least-privilege user. Everything else in this lesson is a detail around those three.

Symptom, cause, fixGarbage outputbaud orframemismatchtrycommonratesmatchbothendsPermissiondeniednot indialoutls -l /dev/ttyUSB0usermod-aGdialoutHang after execexitstatusread tooearlystuck inrecv_exit_statusdrainoutputfirstHost keyacceptedAutoAddPolicyneverfails ahostcheckseed known_hosts, RejectPolicyHalf a messagerecv isnot onemessagefirstrecv isshortframewith alengthprefix
The recurring failures of this lesson, mapped from symptom to cause to fix.
11

Try it

1 min read

The lab has three parts that line up with the three doors. First, open a serial port, or a loopback pair created with socat on the host or com0com on Windows, and exchange length-prefixed frames with pyserial, handling partial reads and timeouts. Second, use paramiko to SFTP a small script to your Pi, execute it, and capture its output and exit code, with host-key verification and key-only login. Third, write a TCP echo client and server pair, then fix the server so it reassembles a partial-read message correctly with the framing helper. The practice pack drills the same skills with shorter questions, including the timeout, framing, and host-key traps.

Key takeaways

  • A UART is asynchronous: each byte is framed by start, data, parity, and stop bits, and both ends must agree on the baud rate and the frame format, 8N1 being the default.
  • A timed-out pyserial read is not an error: it returns whatever arrived, possibly empty; only write_timeout raises.
  • Everything on the wire is bytes: encode text before write and decode with errors=replace after read.
  • SSH-2 authentication is a signature, not an encryption: the client signs with the private key and the server verifies with the public key.
  • Never use AutoAddPolicy in unattended automation; seed known_hosts, keep RejectPolicy, and use key-only login with a least-privilege user.
  • SFTP rides the same SSH connection, and one transport multiplexes channels for exec, shell, and port forwarding.
  • TCP is a byte stream with no message boundaries, so recv can split or merge your messages; frame them with a length prefix or delimiter.
  • Use requests or httpx for plain HTTP and reserve raw sockets for protocols you control; UDP keeps datagrams but guarantees nothing.

Reflect

  1. Explain to a colleague, in two sentences: explain what 8N1 means and how a UART frames each byte with start, data, parity, and stop bits.
  2. Which objective are you least sure about? Re-read that section, then do its exercise in the practice pack.
  3. What surprised you when you ran the commands yourself?

On a real board

Repeat the key commands on your Raspberry Pi or lab board over SSH or the serial console. Note anything that behaves differently from your PC, such as missing tools, a BusyBox shell or a read-only filesystem.

Practise this lessonExercises, quiz, interview preparation and spaced review

References and further reading

Official documentation first. Links checked when the lesson was published.

Find your next step

↑ ↓ to move · Enter to open · Esc to close