commaai/openpilot · critical · TimeoutError

flash did not converge within {FLASH_BUDGET:g}s

Error message

flash did not converge within {FLASH_BUDGET:g}s

What it means

TimeoutError raised by check_budget() in the chestnut firmware flasher when time.monotonic() passes the global deadline set at flash start (FLASH_BUDGET = 600 s). The flasher polls the device through retry loops; each checkpoint calls check_budget so a stuck device cannot loop forever, and this error ends the attempt.

Source

Thrown at openpilot/system/hardware/chestnut/flash.py:42

VBUS_PATH = "/sys/kernel/debug/regulator/smb2-vbus/enable"
IMAGE_OFFSET = 0x100
SECTOR, PAGE = 4096, 128
MAX_CODE_SIZE = 0x10000
FLASH_BUDGET = 600.0
USBDEVFS_CONTROL = 0xC0185500
USBDEVFS_BULK = 0xC0185502
USBDEVFS_SETINTERFACE = 0x80085504
USBDEVFS_SETCONFIGURATION = 0x80045505
USBDEVFS_CLAIMINTERFACE = 0x8004550F
USBDEVFS_RESET = 0x5514
USBDEVFS_CLEAR_HALT = 0x80045515

_deadline = float("inf")


def check_budget():
  if time.monotonic() > _deadline:
    raise TimeoutError(f"flash did not converge within {FLASH_BUDGET:g}s")


class Ctrl(ctypes.Structure):
  _fields_ = [("request_type", ctypes.c_uint8), ("request", ctypes.c_uint8),
              ("value", ctypes.c_uint16), ("index", ctypes.c_uint16),
              ("length", ctypes.c_uint16), ("timeout", ctypes.c_uint32),
              ("data", ctypes.c_void_p)]


class Bulk(ctypes.Structure):
  _fields_ = [("ep", ctypes.c_uint), ("len", ctypes.c_uint),
              ("timeout", ctypes.c_uint), ("data", ctypes.c_void_p)]


class RomFallback(Exception):
  pass

View on GitHub (pinned to 516ec1e682)

Solutions

1. Simply rerun the flasher: a device in the ROM bootloader is normally recovered on the next attempt
2. Use a direct, short USB connection (no hubs) and a different port/cable
3. Check dmesg for USB errors (babble, stall, disconnect) during the failed attempt
4. Confirm only one chestnut device is attached (see find_chestnut)

Example fix

// before
flash_chestnut()  # single attempt, wedges for 600s then times out

// after
for attempt in range(3):
  try:
    flash_chestnut()
    break
  except TimeoutError:
    print(f'flash attempt {attempt} timed out, retrying')

import time

def budget_remaining() -> float:
    from openpilot.system.hardware.chestnut import flash
    return max(0.0, flash._deadline - time.monotonic())

for attempt in range(3):
    try:
        flash_chestnut()
        break
    except TimeoutError:
        print(f'flash timed out on attempt {attempt}; device likely in ROM bootloader, retrying')

When it happens

Trigger: The chestnut device stops responding (no USB re-enumeration after a reset, bulk transfers stalling) so erase/write/verify retries spin until the 600-second budget is exhausted; check_budget then raises at the next call site.

Common situations: Flaky USB cable/port causing stalled transfers, a device wedged in the ROM bootloader, or a hardware fault on the peripheral. Note the device may be left mid-flash (in ROM bootloader mode) — usually recoverable by rerunning.

Related errors

• expected one chestnut, found {len(found)}
• flash controller timeout
• could not disable USB runtime PM: {control}
• chestnut is in use, stop modeld/GPU processes before flashin
• chestnut fell back to the ROM bootloader

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/66e3d228aec98c69. Report an issue: GitHub.