commaai/openpilot · error · EOFError

Compressed {compression} file ended before the end-of-stream

Error message

Compressed {compression} file ended before the end-of-stream marker

What it means

decompress_file() streams the compressed file through a bz2/zstd decompressor, then checks decompressor.eof. If the stream ended without the format's end-of-stream marker, the file is truncated and the partially decompressed output is incomplete, so it raises EOFError naming the compression format. This catches partial/corrupted downloads rather than silently returning a truncated file.

Source

Thrown at openpilot/tools/lib/file_downloader.py:78

def make_decompressor(compression):
  if compression == 'bz2':
    return bz2.BZ2Decompressor()
  if compression == 'zst':
    return zstd.ZstdDecompressor().decompressobj()
  raise ValueError(f"Unsupported compression type: {compression}")


def decompress_file(source, destination, compression=None):
  with open(source, 'rb') as src, open(destination, 'wb') as dst:
    header = src.read(4)
    compression = compression or compression_type(header)
    decompressor = make_decompressor(compression)
    dst.write(decompressor.decompress(header))
    while data := src.read(1024 * 1024):
      dst.write(decompressor.decompress(data))
  if not decompressor.eof:
    raise EOFError(f"Compressed {compression} file ended before the end-of-stream marker")


def materialize_cached_file(source, url, compression):
  local_path = cache_file_path(url, compression)
  if os.path.exists(local_path):
    return local_path

  tmp_fd, tmp_path = tempfile.mkstemp(dir=Paths.download_cache_root())
  os.close(tmp_fd)
  try:
    decompress_file(source, tmp_path, compression)
    shutil.move(tmp_path, local_path)
  except Exception:
    try:
      os.unlink(tmp_path)
    except OSError:
      pass
    raise

View on GitHub (pinned to 516ec1e682)

Solutions

Delete the partial source/destination and re-download the file fully, then retry
Verify size/checksum of the downloaded file against the server (Content-Length) to detect truncation
Clear the download cache entry (cache_file_path) so materialize_cached_file does not reuse a bad artifact

Defensive patterns

Strategy: retry

Validate before calling

expected = remote_size_or_checksum  # from listing/headers
import os
if expected is not None and os.path.getsize(source) != expected:
    raise SystemExit(f"source truncated: {os.path.getsize(source)} != {expected} - re-download")

Try / catch

for attempt in range(3):
    try:
        decompress_file(source, tmp_path, compression)
        break
    except EOFError as e:
        if attempt < 2:
            os.remove(source); re_download(source)
            continue
        raise SystemExit(f"stream truncated after retries: {e}") from e

Prevention

Validate downloaded file size against Content-Length/checksum before decompressing
Clear the cache path from cache_file_path() when a decompress fails, so retries start from a clean state

When it happens

Trigger: decompress_file(source, destination) where source is a bz2 or zst file cut off mid-stream - interrupted download, full disk during fetch, proxy truncation. Note the first 4 header bytes are fed first and the loop continues while data is non-empty; only when the loop ends with eof False does this fire.

Common situations: Cached download interrupted (network drop, Ctrl-C then resumed wrong); log files truncated at upload time; disk quota hit while materializing.

Related errors

AI-assisted analysis of commaai/openpilot@516ec1e682 (2026-08-15). Data as JSON: /api/errors/f8eda19830320e87. Report an issue: GitHub.