{"record":{"id":"995abde1b3b63637","repo":"headroomlabs-ai/headroom","slug":"failed-to-load-tokenizer-for-name-source","errorCode":null,"errorMessage":"failed to load tokenizer for `{name}`: {source}","messagePattern":"failed to load tokenizer for `(.+?)`: (.+?)","errorType":"exception","errorClass":"HfTokenizerError","httpStatus":null,"severity":"error","filePath":"crates/headroom-core/src/tokenizer/hf_impl.rs","lineNumber":38,"sourceCode":"//!\n//! # What's NOT here\n//! - **No tokenizer.json bundled in the binary.** Bundling Llama / Cohere\n//!   tokenizers would add several MB of binary bloat for code paths most users\n//!   don't hit. `from_pretrained` lazily downloads instead.\n\nuse std::path::Path;\nuse std::sync::Arc;\n\nuse thiserror::Error;\nuse tokenizers::Tokenizer as HfInner;\n\nuse super::{Backend, Tokenizer};\n\n#[derive(Debug, Error)]\npub enum HfTokenizerError {\n    /// The bytes / file did not parse as a valid HuggingFace `tokenizer.json`,\n    /// or the model component referenced an unsupported algorithm.\n    #[error(\"failed to load tokenizer for `{name}`: {source}\")]\n    Load {\n        name: String,\n        #[source]\n        source: Box<dyn std::error::Error + Send + Sync>,\n    },\n    /// The HuggingFace Hub fetch failed: network error, 404 on the repo, or\n    /// 401 on a gated model without an `HF_TOKEN`.\n    #[error(\"failed to download `{repo}` from HuggingFace Hub: {source}\")]\n    Hub {\n        repo: String,\n        #[source]\n        source: Box<dyn std::error::Error + Send + Sync>,\n    },\n}\n\n/// Token counter backed by a HuggingFace `tokenizer.json`.\n///\n/// Cheap to clone — internally an `Arc<tokenizers::Tokenizer>`. Construct once","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/crates/headroom-core/src/tokenizer/hf_impl.rs#L20-L56","documentation":"The request declared 'Content-Encoding: deflate' but zlib.decompress() failed (helpers.py:2425). The usual cause is the classic deflate ambiguity: some clients send raw deflate streams while zlib expects a zlib-wrapped stream, producing 'incorrect header check'. Truncated or corrupt bodies raise the same error.","triggerScenarios":"A client sends raw (unwrapped) deflate bytes with 'Content-Encoding: deflate'; a partially uploaded body missing the adler32 trailer; bytes that are actually gzip labeled as deflate.","commonSituations":"HTTP libraries or hand-rolled encoders that use raw deflate (zlib.compressobj with wbits=-15) instead of the zlib wrapper; proxies that mislabel encodings; interrupted uploads retried against a half-consumed stream.","solutions":["Make the sender use zlib-wrapped deflate (python: zlib.compress(data), node: zlib.deflate) rather than raw deflate","Verify standalone: python -c \"import zlib; zlib.decompress(open('body.bin','rb').read())\"","Switch the client to gzip or identity encoding, both of which are unambiguous"],"exampleFix":"# before: raw deflate stream (wbits=-15) labeled 'deflate'\ncomp = zlib.compressobj(9, zlib.DEFLATED, -15)\nbody = comp.compress(data) + comp.flush()\n\n# after: zlib-wrapped deflate\nbody = zlib.compress(data)","handlingStrategy":"validation","validationCode":"# Client-side: confirm the stream is zlib-wrapped (starts with 0x78)\nfirst = compressed_body[0]\nif first != 0x78:\n    raise ValueError(\"raw deflate stream sent with 'deflate' label; use zlib.compress\")","typeGuard":null,"tryCatchPattern":"try:\n    body = await _read_request_body_bytes(request)\nexcept ValueError as exc:\n    return JSONResponse({\"error\": {\"message\": str(exc)}}, status_code=400)","preventionTips":["Prefer zlib.compress over compressobj with negative wbits for HTTP bodies","Prefer gzip/identity over deflate to sidestep the raw-vs-zlib ambiguity","Pin the compression decision in one shared client function"],"tags":["compression","deflate","zlib","corrupt-body","request-body"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}