commaai/openpilot · critical · RuntimeError

final full-image verification failed

Error message

final full-image verification failed

What it means

After programming all changed sectors, write_image() does a stable_read() (three passes) of the full span and compares it to the target image. If the read-back still differs, the flash verify failed and the device is left unverified. This catches programming failures, flaky USB/serial links, and flash cells that do not hold what was written.

Source

Thrown at openpilot/system/hardware/chestnut/flash.py:553

    current = stable_read(flash, first_sector, span - first_sector)
    target = bytearray(current)
    target[:len(config)] = config
    target[IMAGE_OFFSET - first_sector:image_end - first_sector] = image
    target = bytes(target)
    print(f"target {len(image)} bytes at 0x{IMAGE_OFFSET:05x}, sha256={hashlib.sha256(image).hexdigest()}", flush=True)

    for addr in range(first_sector, span, SECTOR):
      off = addr - first_sector
      wanted = target[off:off + SECTOR]
      if current[off:off + SECTOR] == wanted:
        print(f"sector 0x{addr:05x}: unchanged", flush=True)
      else:
        print(f"sector 0x{addr:05x}: programming", flush=True)
        program_sector(flash, addr, wanted)

    verified = stable_read(flash, first_sector, span - first_sector, 3)
    if verified != target:
      raise RuntimeError("final full-image verification failed")
    print(f"verified sha256={hashlib.sha256(verified).hexdigest()}", flush=True)
  finally:
    flash.close()

  activate(expected_product)


def main():
  parser = argparse.ArgumentParser(description="check and flash the bundled chestnut firmware")
  parser.add_argument("version", nargs="?", help="expected firmware version hash")
  parser.add_argument("--force", action="store_true", help="reflash even when the version matches")
  args = parser.parse_args()
  if os.geteuid() != 0:
    raise RuntimeError("flash.py must run as root")
  flash_chestnut(expected_version=args.version, force=args.force)


if __name__ == "__main__":

View on GitHub (pinned to 516ec1e682)

Solutions

1. Check/replace the USB cable and port, power the device from a stable supply, then re-run flash.py --force
2. Re-run with --force to reprogram: single-bit programming glitches often clear on a second pass
3. Look at the per-sector 'programming' vs 'unchanged' output above the error to see if failures cluster at specific addresses - recurring failures at the same sector indicate a worn flash block
4. If verification keeps failing on the same sectors, the device hardware is suspect - escalate to comma support

for attempt in range(2):
    try:
        write_image(image, expected_product, product, force=True)
        break
    except RuntimeError as e:
        if 'verification failed' in str(e) and attempt == 0:
            print("verify failed once; re-seating connections and retrying")
            continue
        raise

When it happens

Trigger: A sector program silently corrupted data (bad flash block, voltage dip, loose cable); stable_read returned inconsistent data across its 3 attempts due to a noisy connection; the image changed on disk between the initial read and verification (rare re-entrancy).

Common situations: Marginal USB cable or powered hub during a long flash; failing flash chip on the chestnut; EMI or undervoltage on the device during programming.

Related errors

• flash did not converge within {FLASH_BUDGET:g}s
• expected one chestnut, found {len(found)}
• bundled firmware is {expected_product!r}, expected version {
• Unhandled sparse chunk type
• Raw hash mismatch '{raw_hash.hexdigest().lower()}'

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/d866f31445647128. Report an issue: GitHub.