{"record":{"id":"e6539995243d933b","repo":"commaai/openpilot","slug":"flash-controller-timeout","errorCode":null,"errorMessage":"flash controller timeout","messagePattern":"flash controller timeout","errorType":"console","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"openpilot/system/hardware/chestnut/flash.py","lineNumber":187,"sourceCode":"    fcntl.ioctl(self.fd, USBDEVFS_CONTROL,\n                Ctrl(0x40, 0xE5, addr & 0xFFFF, value & 0xFFFF, 0, 2000, None))\n\n  def reg_read(self, addr, length=1):\n    buf = (ctypes.c_ubyte * length)()\n    fcntl.ioctl(self.fd, USBDEVFS_CONTROL,\n                Ctrl(0xC0, 0xE4, addr & 0xFFFF, 0, length, 2000, ctypes.cast(buf, ctypes.c_void_p)))\n    return bytes(buf)\n\n  def write_buffer(self, data):\n    for i, value in enumerate(data):\n      self.reg_write(0x7000 + i, value)\n\n  def wait_controller(self, timeout=2.0):\n    deadline = time.monotonic() + timeout\n    while time.monotonic() < deadline:\n      if not self.reg_read(0xC8A9)[0] & 1:\n        return\n    raise TimeoutError(\"flash controller timeout\")\n\n  def transaction(self, command, addr=0, length=0, addr_len=0x07, mode=0):\n    for reg, value in ((0xC8AD, mode), (0xC8AE, 0), (0xC8AF, 0), (0xC8AA, command), (0xC8AC, addr_len),\n                       (0xC8A1, addr), (0xC8A2, addr >> 8), (0xC8AB, addr >> 16), (0xC8A3, length >> 8), (0xC8A4, length)):\n      self.reg_write(reg, value & 0xFF)\n    self.reg_write(0xC8A9, 1)\n    self.wait_controller()\n    for _ in range(4):\n      self.reg_write(0xC8AD, 0)\n\n  def write_enable(self):\n    for reg, value in ((0xC8AD, 0), (0xC8AA, 0x06), (0xC8AC, 0x04), (0xC8A3, 0), (0xC8A4, 0), (0xC8A9, 1)):\n      self.reg_write(reg, value)\n    self.wait_controller()\n\n  def status(self):\n    self.transaction(0x05, length=1, addr_len=0x04)\n    return self.reg_read(0x7000)[0]","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/commaai/openpilot/blob/516ec1e68203439a73f340f1d0b3b91eabc626ee/openpilot/system/hardware/chestnut/flash.py#L169-L205","documentation":"TimeoutError raised by Flash.wait_controller() (default timeout 2.0s): after a transaction is kicked off by writing 1 to register 0xC8A9, the controller busy bit (bit 0 read back via reg_read(0xC8A9)) never cleared. The SPI controller inside the ASM2464 stalled mid-command - typically because the USB link dropped, the device reset, or the controller got a malformed register sequence.","triggerScenarios":"Any Flash.transaction(), write_enable(), or init() call: reg_write(0xC8A9, 1) starts the command, then wait_controller() polls. Fires when the device stops responding over EP0 control transfers (busy bit stays set or reg_read returns stale data), e.g. after the enclosure lost power or runtime-PM suspended the controller mid-poll.","commonSituations":"USB runtime PM kicking in because power/control was not 'on' (disable_runtime_pm failed or was skipped); flaky USB-C connection; device watchdog reset mid-transaction; polling after the device was unplugged.","solutions":["The retry layer (with_retries + reconnect) usually handles this - make sure transient operations go through with_retries, which reconnects (re-claims the interface, re-inits) and retries","Verify power/control is 'on' and autosuspend is disabled for the device in /sys/bus/usb/devices/<dev>/power/ before flashing","Check the physical link (cable, port, VBUS per /sys/kernel/debug/regulator/smb2-vbus/enable) and reseat if unstable","If it reproduces deterministically at the same step, capture dmesg for usb disconnects and check for an actual device reset mid-command"],"exampleFix":"# before\nflash.transaction(0x03, 0, 4096)  # bare call, one stall kills the run\n\n# after: route through the retry/reconnect helper\nwith_retries(flash, 'read 0x00000', lambda: flash.transaction(0x03, 0, 4096))","handlingStrategy":"retry","validationCode":"import ctypes, fcntl, os\nfrom flash import find_chestnut, open_device, Ctrl, USBDEVFS_CONTROL\n\ndef controller_alive() -> bool:\n    path, _, _ = find_chestnut()\n    if path is None:\n        return False\n    fd = open_device(path)\n    try:\n        buf = (ctypes.c_ubyte * 1)()\n        fcntl.ioctl(fd, USBDEVFS_CONTROL,\n                    Ctrl(0xC0, 0xE4, 0xC8A9, 0, 1, 2000, ctypes.cast(buf, ctypes.c_void_p)))\n        return True\n    except OSError:\n        return False\n    finally:\n        os.close(fd)","typeGuard":null,"tryCatchPattern":"try:\n    flash.transaction(0x03, addr, n)\nexcept TimeoutError as e:\n    if 'flash controller' in str(e):\n        reconnect(flash)\n        retry_operation()","preventionTips":["Disable USB runtime PM for the device before long flash sessions","Route every controller transaction through with_retries, never call it bare","Watch dmesg for disconnect spam during a session - abort early rather than fight timeouts"],"tags":["spi","timeout","usb","hardware","flashing"],"backgroundTag":null,"analyzedSha":"516ec1e68203439a73f340f1d0b3b91eabc626ee","analyzedAt":"2026-08-15T00:17:37.461Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}