{"record":{"id":"f8eda19830320e87","repo":"commaai/openpilot","slug":"compressed-compression-file-ended-before-the-end","errorCode":null,"errorMessage":"Compressed {compression} file ended before the end-of-stream marker","messagePattern":"Compressed (.+?) file ended before the end-of-stream marker","errorType":"exception","errorClass":"EOFError","httpStatus":null,"severity":"error","filePath":"openpilot/tools/lib/file_downloader.py","lineNumber":78,"sourceCode":"\ndef make_decompressor(compression):\n  if compression == 'bz2':\n    return bz2.BZ2Decompressor()\n  if compression == 'zst':\n    return zstd.ZstdDecompressor().decompressobj()\n  raise ValueError(f\"Unsupported compression type: {compression}\")\n\n\ndef decompress_file(source, destination, compression=None):\n  with open(source, 'rb') as src, open(destination, 'wb') as dst:\n    header = src.read(4)\n    compression = compression or compression_type(header)\n    decompressor = make_decompressor(compression)\n    dst.write(decompressor.decompress(header))\n    while data := src.read(1024 * 1024):\n      dst.write(decompressor.decompress(data))\n  if not decompressor.eof:\n    raise EOFError(f\"Compressed {compression} file ended before the end-of-stream marker\")\n\n\ndef materialize_cached_file(source, url, compression):\n  local_path = cache_file_path(url, compression)\n  if os.path.exists(local_path):\n    return local_path\n\n  tmp_fd, tmp_path = tempfile.mkstemp(dir=Paths.download_cache_root())\n  os.close(tmp_fd)\n  try:\n    decompress_file(source, tmp_path, compression)\n    shutil.move(tmp_path, local_path)\n  except Exception:\n    try:\n      os.unlink(tmp_path)\n    except OSError:\n      pass\n    raise","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/commaai/openpilot/blob/516ec1e68203439a73f340f1d0b3b91eabc626ee/openpilot/tools/lib/file_downloader.py#L60-L96","documentation":"decompress_file() streams the compressed file through a bz2/zstd decompressor, then checks decompressor.eof. If the stream ended without the format's end-of-stream marker, the file is truncated and the partially decompressed output is incomplete, so it raises EOFError naming the compression format. This catches partial/corrupted downloads rather than silently returning a truncated file.","triggerScenarios":"decompress_file(source, destination) where source is a bz2 or zst file cut off mid-stream - interrupted download, full disk during fetch, proxy truncation. Note the first 4 header bytes are fed first and the loop continues while data is non-empty; only when the loop ends with eof False does this fire.","commonSituations":"Cached download interrupted (network drop, Ctrl-C then resumed wrong); log files truncated at upload time; disk quota hit while materializing.","solutions":["Delete the partial source/destination and re-download the file fully, then retry","Verify size/checksum of the downloaded file against the server (Content-Length) to detect truncation","Clear the download cache entry (cache_file_path) so materialize_cached_file does not reuse a bad artifact"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"expected = remote_size_or_checksum  # from listing/headers\nimport os\nif expected is not None and os.path.getsize(source) != expected:\n    raise SystemExit(f\"source truncated: {os.path.getsize(source)} != {expected} - re-download\")","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        decompress_file(source, tmp_path, compression)\n        break\n    except EOFError as e:\n        if attempt < 2:\n            os.remove(source); re_download(source)\n            continue\n        raise SystemExit(f\"stream truncated after retries: {e}\") from e","preventionTips":["Validate downloaded file size against Content-Length/checksum before decompressing","Clear the cache path from cache_file_path() when a decompress fails, so retries start from a clean state"],"tags":["compression","truncated-file","download","openpilot"],"backgroundTag":null,"analyzedSha":"516ec1e68203439a73f340f1d0b3b91eabc626ee","analyzedAt":"2026-08-15T00:17:37.461Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}