commaai/openpilot · warning · RuntimeError

sector verification failed

Error message

sector verification failed

What it means

RuntimeError inside program_sector(): the final whole-sector read-back (all 4096 bytes) does not equal the target image, even though every page individually verified. This catches errors that only appear on a full re-read: read instability across time, or a device glitch between the per-page loop and the final check. It also catches skipped all-0xFF pages whose on-flash content differs.

Source

Thrown at openpilot/system/hardware/chestnut/flash.py:320

    if any(x != reads[0] for x in reads[1:]):
      raise RuntimeError(f"unstable flash read at 0x{addr:05x}")
    return reads[0]
  return with_retries(flash, f"read 0x{addr:05x}", read)


def program_sector(flash, addr, target):
  def program():
    flash.erase_sector(addr)
    if flash.read(addr, SECTOR) != bytes([0xFF]) * SECTOR:
      raise RuntimeError("sector erase verification failed")
    for off in range(0, SECTOR, PAGE):
      chunk = target[off:off + PAGE]
      if chunk != bytes([0xFF]) * len(chunk):
        flash.program(addr + off, chunk)
        if flash.read(addr + off, len(chunk)) != chunk:
          raise RuntimeError(f"page verify failed at 0x{addr + off:05x}")
    if flash.read(addr, SECTOR) != target:
      raise RuntimeError("sector verification failed")
  with_retries(flash, f"sector 0x{addr:05x}", program)


def config_path():
  return os.path.join(CONFIG_DIR, f"{os.uname().nodename}.bin")


def saved_config(path, data):
  os.makedirs(os.path.dirname(path), exist_ok=True)
  try:
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
  except FileExistsError as e:
    backup = open(path, "rb").read()
    if len(backup) != 0x100:
      raise RuntimeError(f"invalid config backup: {path}") from e
    if backup != data:
      print(f"restoring config from {path}", flush=True)
    return backup

View on GitHub (pinned to 516ec1e682)

Solutions

1. Let with_retries retry the sector - transient instability is the designed target
2. If it fails repeatedly at the same sector, diff flash.read output against the target byte-by-byte to see whether it is skipped-page content (erase issue) or random bits (link issue)
3. Stabilize the physical link before further attempts; each retry costs budget (FLASH_BUDGET is 600s total)
4. If the time budget expires during retries, the tool aborts - fix the root cause and restart rather than fighting the budget

try:
    program_sector(flash, addr, target)
except TimeoutError as e:
    if 'converge' in str(e):
        raise  # retry budget spent; investigate the physical layer

When it happens

Trigger: The last statement of program_sector()'s program(): flash.read(addr, SECTOR) != target. Occurs when the device glitches between the page loop and the sector re-read, or when pages skipped as all-0xFF in the target are not actually 0xFF on flash.

Common situations: Intermittent USB corruption hitting the final long read (4096 bytes = more exposure than per-page reads); a device dropping off the bus mid-sector; erase check passed but region reads unreliably afterwards.

Related errors

• sector erase verification failed
• page verify failed at 0x{addr + off:05x}
• SPI flash WIP timeout
• unstable flash read at 0x{addr:05x}
• {func.__name__} failed after retry

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/6bba519c4746710c. Report an issue: GitHub.