headroomlabs-ai/headroom · error · HfTokenizerError

failed to load tokenizer for `{name}`: {source}

Error message

failed to load tokenizer for `{name}`: {source}

What it means

The request declared 'Content-Encoding: deflate' but zlib.decompress() failed (helpers.py:2425). The usual cause is the classic deflate ambiguity: some clients send raw deflate streams while zlib expects a zlib-wrapped stream, producing 'incorrect header check'. Truncated or corrupt bodies raise the same error.

Source

Thrown at crates/headroom-core/src/tokenizer/hf_impl.rs:38

//!
//! # What's NOT here
//! - **No tokenizer.json bundled in the binary.** Bundling Llama / Cohere
//!   tokenizers would add several MB of binary bloat for code paths most users
//!   don't hit. `from_pretrained` lazily downloads instead.

use std::path::Path;
use std::sync::Arc;

use thiserror::Error;
use tokenizers::Tokenizer as HfInner;

use super::{Backend, Tokenizer};

#[derive(Debug, Error)]
pub enum HfTokenizerError {
    /// The bytes / file did not parse as a valid HuggingFace `tokenizer.json`,
    /// or the model component referenced an unsupported algorithm.
    #[error("failed to load tokenizer for `{name}`: {source}")]
    Load {
        name: String,
        #[source]
        source: Box<dyn std::error::Error + Send + Sync>,
    },
    /// The HuggingFace Hub fetch failed: network error, 404 on the repo, or
    /// 401 on a gated model without an `HF_TOKEN`.
    #[error("failed to download `{repo}` from HuggingFace Hub: {source}")]
    Hub {
        repo: String,
        #[source]
        source: Box<dyn std::error::Error + Send + Sync>,
    },
}

/// Token counter backed by a HuggingFace `tokenizer.json`.
///
/// Cheap to clone — internally an `Arc<tokenizers::Tokenizer>`. Construct once

View on GitHub (pinned to 322425c43b)

Solutions

1. Make the sender use zlib-wrapped deflate (python: zlib.compress(data), node: zlib.deflate) rather than raw deflate
2. Verify standalone: python -c "import zlib; zlib.decompress(open('body.bin','rb').read())"
3. Switch the client to gzip or identity encoding, both of which are unambiguous

Example fix

# before: raw deflate stream (wbits=-15) labeled 'deflate'
comp = zlib.compressobj(9, zlib.DEFLATED, -15)
body = comp.compress(data) + comp.flush()

# after: zlib-wrapped deflate
body = zlib.compress(data)

# Client-side: confirm the stream is zlib-wrapped (starts with 0x78)
first = compressed_body[0]
if first != 0x78:
    raise ValueError("raw deflate stream sent with 'deflate' label; use zlib.compress")

try:
    body = await _read_request_body_bytes(request)
except ValueError as exc:
    return JSONResponse({"error": {"message": str(exc)}}, status_code=400)

When it happens

Trigger: A client sends raw (unwrapped) deflate bytes with 'Content-Encoding: deflate'; a partially uploaded body missing the adler32 trailer; bytes that are actually gzip labeled as deflate.

Common situations: HTTP libraries or hand-rolled encoders that use raw deflate (zlib.compressobj with wbits=-15) instead of the zlib wrapper; proxies that mislabel encodings; interrupted uploads retried against a half-consumed stream.

Related errors

• ccr redis backend init failed: {0}
• ccr backend `{backend}` is not compiled in; rebuild with `--
• magika session init failed: {0}
• ccr sqlite backend init failed: {0}
• failed to download `{repo}` from HuggingFace Hub: {source}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/995abde1b3b63637. Report an issue: GitHub.