What causes "unstable flash read at 0x{addr:05x}" in commaai/openpilot?

Let with_retries do its job first - a single unstable read recovers via reconnect; only persistent instability matters

commaai/openpilot · warning · RuntimeError

unstable flash read at 0x{addr:05x}

Error message

unstable flash read at 0x{addr:05x}

What it means

RuntimeError raised inside stable_read(): the same address range was read count times (default 2) back-to-back and the results differed. SPI NOR reads are deterministic, so disagreement means the read path is unreliable - flaky USB link, a controller mid-reset, or a device that re-enumerated between reads. It is raised deliberately so with_retries catches it, reconnects, and retries the whole read set.

Source

Thrown at openpilot/system/hardware/chestnut/flash.py:303

def with_retries(flash, label, operation):
  # on any transfer error, reconnect and restart the operation
  attempt = 0
  while True:
    attempt += 1
    try:
      return operation()
    except (OSError, TimeoutError, RuntimeError) as e:
      check_budget()
      print(f"{label} attempt {attempt}: {e}", flush=True)
      reconnect(flash)


def stable_read(flash, addr, length, count=2):
  def read():
    reads = [flash.read(addr, length) for _ in range(count)]
    if any(x != reads[0] for x in reads[1:]):
      raise RuntimeError(f"unstable flash read at 0x{addr:05x}")
    return reads[0]
  return with_retries(flash, f"read 0x{addr:05x}", read)


def program_sector(flash, addr, target):
  def program():
    flash.erase_sector(addr)
    if flash.read(addr, SECTOR) != bytes([0xFF]) * SECTOR:
      raise RuntimeError("sector erase verification failed")
    for off in range(0, SECTOR, PAGE):
      chunk = target[off:off + PAGE]
      if chunk != bytes([0xFF]) * len(chunk):
        flash.program(addr + off, chunk)
        if flash.read(addr + off, len(chunk)) != chunk:
          raise RuntimeError(f"page verify failed at 0x{addr + off:05x}")
    if flash.read(addr, SECTOR) != target:
      raise RuntimeError("sector verification failed")
  with_retries(flash, f"sector 0x{addr:05x}", program)

View on GitHub (pinned to 516ec1e682)

Solutions

Let with_retries do its job first - a single unstable read recovers via reconnect; only persistent instability matters
Harden the link: shorter/better USB-C cable, direct host-controller port, disable runtime PM for the device
Increase the read count (count=3) to distinguish one-off glitches from persistent corruption
If unstable reads persist across reconnects on every sector, stop flashing and fix the physical layer - continuing risks programming garbage and bricking into ROM mode

Example fix

# before
data = flash.read(addr, length)  # unverified

# after
data = stable_read(flash, addr, length, count=2)

Defensive patterns

Strategy: retry

Try / catch

# stable_read already wraps this in with_retries; only handle the terminal case
try:
    data = stable_read(flash, addr, length)
except (TimeoutError, RuntimeError):
    abort_session('bus unstable - fix physical layer')

Prevention

Always read critical regions through stable_read, never bare flash.read
Keep cables short and seated; avoid hubs during flashing
Treat any unstable read as a warning about the whole session, not one address

When it happens

Trigger: stable_read(flash, addr, length) during read-back verification of the config (0x100 region) or firmware regions: two consecutive flash.read() calls return different bytes. Typical when the EP0 control-transfer link corrupts data or the device glitches - marginal USB-C connection, runtime PM transitions, or another process touching the device.

Common situations: Cheap or flaky USB-C cable or dock; EMI-heavy environment; the enclosure's port losing signal integrity; reads racing a device reset; symptoms are intermittent and move between addresses.

Related errors

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/e31f9a83df256e2d. Report an issue: GitHub.