{"record":{"id":"6bba519c4746710c","repo":"commaai/openpilot","slug":"sector-verification-failed","errorCode":null,"errorMessage":"sector verification failed","messagePattern":"sector verification failed","errorType":"console","errorClass":"RuntimeError","httpStatus":null,"severity":"warning","filePath":"openpilot/system/hardware/chestnut/flash.py","lineNumber":320,"sourceCode":"    if any(x != reads[0] for x in reads[1:]):\n      raise RuntimeError(f\"unstable flash read at 0x{addr:05x}\")\n    return reads[0]\n  return with_retries(flash, f\"read 0x{addr:05x}\", read)\n\n\ndef program_sector(flash, addr, target):\n  def program():\n    flash.erase_sector(addr)\n    if flash.read(addr, SECTOR) != bytes([0xFF]) * SECTOR:\n      raise RuntimeError(\"sector erase verification failed\")\n    for off in range(0, SECTOR, PAGE):\n      chunk = target[off:off + PAGE]\n      if chunk != bytes([0xFF]) * len(chunk):\n        flash.program(addr + off, chunk)\n        if flash.read(addr + off, len(chunk)) != chunk:\n          raise RuntimeError(f\"page verify failed at 0x{addr + off:05x}\")\n    if flash.read(addr, SECTOR) != target:\n      raise RuntimeError(\"sector verification failed\")\n  with_retries(flash, f\"sector 0x{addr:05x}\", program)\n\n\ndef config_path():\n  return os.path.join(CONFIG_DIR, f\"{os.uname().nodename}.bin\")\n\n\ndef saved_config(path, data):\n  os.makedirs(os.path.dirname(path), exist_ok=True)\n  try:\n    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)\n  except FileExistsError as e:\n    backup = open(path, \"rb\").read()\n    if len(backup) != 0x100:\n      raise RuntimeError(f\"invalid config backup: {path}\") from e\n    if backup != data:\n      print(f\"restoring config from {path}\", flush=True)\n    return backup","sourceCodeStart":302,"sourceCodeEnd":338,"githubUrl":"https://github.com/commaai/openpilot/blob/516ec1e68203439a73f340f1d0b3b91eabc626ee/openpilot/system/hardware/chestnut/flash.py#L302-L338","documentation":"RuntimeError inside program_sector(): the final whole-sector read-back (all 4096 bytes) does not equal the target image, even though every page individually verified. This catches errors that only appear on a full re-read: read instability across time, or a device glitch between the per-page loop and the final check. It also catches skipped all-0xFF pages whose on-flash content differs.","triggerScenarios":"The last statement of program_sector()'s program(): flash.read(addr, SECTOR) != target. Occurs when the device glitches between the page loop and the sector re-read, or when pages skipped as all-0xFF in the target are not actually 0xFF on flash.","commonSituations":"Intermittent USB corruption hitting the final long read (4096 bytes = more exposure than per-page reads); a device dropping off the bus mid-sector; erase check passed but region reads unreliably afterwards.","solutions":["Let with_retries retry the sector - transient instability is the designed target","If it fails repeatedly at the same sector, diff flash.read output against the target byte-by-byte to see whether it is skipped-page content (erase issue) or random bits (link issue)","Stabilize the physical link before further attempts; each retry costs budget (FLASH_BUDGET is 600s total)","If the time budget expires during retries, the tool aborts - fix the root cause and restart rather than fighting the budget"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    program_sector(flash, addr, target)\nexcept TimeoutError as e:\n    if 'converge' in str(e):\n        raise  # retry budget spent; investigate the physical layer","preventionTips":["Respect the 600s FLASH_BUDGET: fix instability early instead of burning retries","Log which sectors needed retries - clusters indicate hardware","After a retried session, do a final full-image stable_read comparison"],"tags":["spi","flash-write","verify","retry","sector"],"backgroundTag":null,"analyzedSha":"516ec1e68203439a73f340f1d0b3b91eabc626ee","analyzedAt":"2026-08-15T00:17:37.461Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}