commaai/openpilot · error · TimeoutError

flash controller timeout

Error message

flash controller timeout

What it means

TimeoutError raised by Flash.wait_controller() (default timeout 2.0s): after a transaction is kicked off by writing 1 to register 0xC8A9, the controller busy bit (bit 0 read back via reg_read(0xC8A9)) never cleared. The SPI controller inside the ASM2464 stalled mid-command - typically because the USB link dropped, the device reset, or the controller got a malformed register sequence.

Source

Thrown at openpilot/system/hardware/chestnut/flash.py:187

    fcntl.ioctl(self.fd, USBDEVFS_CONTROL,
                Ctrl(0x40, 0xE5, addr & 0xFFFF, value & 0xFFFF, 0, 2000, None))

  def reg_read(self, addr, length=1):
    buf = (ctypes.c_ubyte * length)()
    fcntl.ioctl(self.fd, USBDEVFS_CONTROL,
                Ctrl(0xC0, 0xE4, addr & 0xFFFF, 0, length, 2000, ctypes.cast(buf, ctypes.c_void_p)))
    return bytes(buf)

  def write_buffer(self, data):
    for i, value in enumerate(data):
      self.reg_write(0x7000 + i, value)

  def wait_controller(self, timeout=2.0):
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
      if not self.reg_read(0xC8A9)[0] & 1:
        return
    raise TimeoutError("flash controller timeout")

  def transaction(self, command, addr=0, length=0, addr_len=0x07, mode=0):
    for reg, value in ((0xC8AD, mode), (0xC8AE, 0), (0xC8AF, 0), (0xC8AA, command), (0xC8AC, addr_len),
                       (0xC8A1, addr), (0xC8A2, addr >> 8), (0xC8AB, addr >> 16), (0xC8A3, length >> 8), (0xC8A4, length)):
      self.reg_write(reg, value & 0xFF)
    self.reg_write(0xC8A9, 1)
    self.wait_controller()
    for _ in range(4):
      self.reg_write(0xC8AD, 0)

  def write_enable(self):
    for reg, value in ((0xC8AD, 0), (0xC8AA, 0x06), (0xC8AC, 0x04), (0xC8A3, 0), (0xC8A4, 0), (0xC8A9, 1)):
      self.reg_write(reg, value)
    self.wait_controller()

  def status(self):
    self.transaction(0x05, length=1, addr_len=0x04)
    return self.reg_read(0x7000)[0]

View on GitHub (pinned to 516ec1e682)

Solutions

1. The retry layer (with_retries + reconnect) usually handles this - make sure transient operations go through with_retries, which reconnects (re-claims the interface, re-inits) and retries
2. Verify power/control is 'on' and autosuspend is disabled for the device in /sys/bus/usb/devices/<dev>/power/ before flashing
3. Check the physical link (cable, port, VBUS per /sys/kernel/debug/regulator/smb2-vbus/enable) and reseat if unstable
4. If it reproduces deterministically at the same step, capture dmesg for usb disconnects and check for an actual device reset mid-command

Example fix

# before
flash.transaction(0x03, 0, 4096)  # bare call, one stall kills the run

# after: route through the retry/reconnect helper
with_retries(flash, 'read 0x00000', lambda: flash.transaction(0x03, 0, 4096))

import ctypes, fcntl, os
from flash import find_chestnut, open_device, Ctrl, USBDEVFS_CONTROL

def controller_alive() -> bool:
    path, _, _ = find_chestnut()
    if path is None:
        return False
    fd = open_device(path)
    try:
        buf = (ctypes.c_ubyte * 1)()
        fcntl.ioctl(fd, USBDEVFS_CONTROL,
                    Ctrl(0xC0, 0xE4, 0xC8A9, 0, 1, 2000, ctypes.cast(buf, ctypes.c_void_p)))
        return True
    except OSError:
        return False
    finally:
        os.close(fd)

try:
    flash.transaction(0x03, addr, n)
except TimeoutError as e:
    if 'flash controller' in str(e):
        reconnect(flash)
        retry_operation()

When it happens

Trigger: Any Flash.transaction(), write_enable(), or init() call: reg_write(0xC8A9, 1) starts the command, then wait_controller() polls. Fires when the device stops responding over EP0 control transfers (busy bit stays set or reg_read returns stale data), e.g. after the enclosure lost power or runtime-PM suspended the controller mid-poll.

Common situations: USB runtime PM kicking in because power/control was not 'on' (disable_runtime_pm failed or was skipped); flaky USB-C connection; device watchdog reset mid-transaction; polling after the device was unplugged.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• flash did not converge within {FLASH_BUDGET:g}s
• SPI flash WIP timeout
• expected one chestnut, found {len(found)}
• chestnut is in use, stop modeld/GPU processes before flashin
• chestnut did not enumerate within {timeout:g}s

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/e6539995243d933b. Report an issue: GitHub.