{"record":{"id":"4f53d589bd9aac78","repo":"commaai/openpilot","slug":"unsupported-compression-type-compression","errorCode":null,"errorMessage":"Unsupported compression type: {compression}","messagePattern":"Unsupported compression type: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"openpilot/tools/lib/file_downloader.py","lineNumber":66,"sourceCode":"  if compression:\n    url_without_query = f\"decompressed-{compression}:{url_without_query}\"\n  return os.path.join(Paths.download_cache_root(), hashlib.sha256(url_without_query.encode()).hexdigest())\n\n\ndef compression_type(data):\n  if data.startswith(b'BZh'):\n    return 'bz2'\n  if data.startswith(b'\\x28\\xb5\\x2f\\xfd'):\n    return 'zst'\n  return None\n\n\ndef make_decompressor(compression):\n  if compression == 'bz2':\n    return bz2.BZ2Decompressor()\n  if compression == 'zst':\n    return zstd.ZstdDecompressor().decompressobj()\n  raise ValueError(f\"Unsupported compression type: {compression}\")\n\n\ndef decompress_file(source, destination, compression=None):\n  with open(source, 'rb') as src, open(destination, 'wb') as dst:\n    header = src.read(4)\n    compression = compression or compression_type(header)\n    decompressor = make_decompressor(compression)\n    dst.write(decompressor.decompress(header))\n    while data := src.read(1024 * 1024):\n      dst.write(decompressor.decompress(data))\n  if not decompressor.eof:\n    raise EOFError(f\"Compressed {compression} file ended before the end-of-stream marker\")\n\n\ndef materialize_cached_file(source, url, compression):\n  local_path = cache_file_path(url, compression)\n  if os.path.exists(local_path):\n    return local_path","sourceCodeStart":48,"sourceCodeEnd":84,"githubUrl":"https://github.com/commaai/openpilot/blob/516ec1e68203439a73f340f1d0b3b91eabc626ee/openpilot/tools/lib/file_downloader.py#L48-L84","documentation":"make_decompressor() only accepts 'bz2' and 'zst'. decompress_file() passes either the caller-supplied compression argument or the value sniffed from the 4-byte header (compression_type returns None for unknown magic). A None or unknown value reaching make_decompressor raises this ValueError, meaning the caller forced a bad compression name or the file magic was unrecognized and None propagated.","triggerScenarios":"Calling decompress_file(..., compression='gzip') or 'xz'; passing compression=None on a file whose first 4 bytes are neither BZh ('BZh') nor the zstd magic 0x28B52FFD; corrupt header bytes.","commonSituations":"Server starts serving gzip/plain data while the client assumed bz2; truncated download losing the magic bytes; hardcoded compression name drifting from actual format.","solutions":["If you know the true format, pass it explicitly: decompress_file(src, dst, compression='bz2') - but only bz2/zst are supported","For gzip/xz, decompress outside this helper (gzip module / lzma module) - it deliberately supports only bz2 and zstd","Check the first 4 bytes of the file (xxd | head) to see whether the magic matches what you claimed"],"exampleFix":"# before\ndecompress_file(src, dst, compression='gzip')\n\n# after (gzip is unsupported here - use the stdlib)\nimport gzip\nwith gzip.open(src, 'rb') as f_in, open(dst, 'wb') as f_out:\n    shutil.copyfileobj(f_in, f_out)","handlingStrategy":"validation","validationCode":"comp = compression if compression else compression_type(open(source, 'rb').read(4))\nassert comp in ('bz2', 'zst'), f\"unsupported/undetected compression {comp!r}; this helper handles only bz2 and zst\"","typeGuard":"def is_supported_compression(name: str | None) -> bool:\n    \"\"\"True when name is one of the decompressor's supported formats.\"\"\"\n    return name in ('bz2', 'zst')","tryCatchPattern":"try:\n    decompress_file(src, dst, compression=comp)\nexcept ValueError as e:\n    if 'Unsupported compression' in str(e):\n        raise SystemExit(f'{comp} not supported; use gzip/lzma modules for other formats')\n    raise","preventionTips":["Sniff the 4-byte magic yourself and route gzip/xz/plain files to the appropriate stdlib module","Never hardcode a compression name without verifying the file's magic bytes first"],"tags":["compression","decompression","validation","openpilot"],"backgroundTag":null,"analyzedSha":"516ec1e68203439a73f340f1d0b3b91eabc626ee","analyzedAt":"2026-08-15T00:17:37.461Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}