headroomlabs-ai/headroom · warning · NotImplementedError

Decoding not available for {self.model} - tokenizer {self.to

Error message

Decoding not available for {self.model} - tokenizer {self.tokenizer_name} could not be loaded

What it means

HuggingFaceTokenizer.decode() raises NotImplementedError when the transformers tokenizer failed to load and the instance is in estimation-fallback mode (checked via _use_fallback()). The message identifies the model and tokenizer name so you can tell which asset failed. Counting still works; token<->text round-trips do not.

Source

Thrown at headroom/tokenizers/huggingface.py:384

                f"Encoding not available for {self.model} - "
                f"tokenizer {self.tokenizer_name} could not be loaded"
            )
        return self.tokenizer.encode(text, add_special_tokens=False)

    def decode(self, tokens: list[int]) -> str:
        """Decode token IDs to text.

        Args:
            tokens: List of token IDs.

        Returns:
            Decoded text.

        Raises:
            NotImplementedError: If tokenizer not available.
        """
        if self._use_fallback():
            raise NotImplementedError(
                f"Decoding not available for {self.model} - "
                f"tokenizer {self.tokenizer_name} could not be loaded"
            )
        return self.tokenizer.decode(tokens)

    @classmethod
    def is_available(cls) -> bool:
        """Check if HuggingFace tokenizers are available.

        Returns:
            True if transformers is installed.
        """
        try:
            import transformers  # noqa: F401

            return True
        except ImportError:
            return False

View on GitHub (pinned to 322425c43b)

Solutions

Restore the load: valid model id, credentials for gated models (HF_TOKEN), reachable hub or pre-populated HF_HOME cache.
Validate availability before decoding: HuggingFaceTokenizer.is_available() plus a check that _use_fallback() is False (or attempt a tiny encode as a probe).
Catch NotImplementedError and fall back to count-only logic if round-tripping is optional.

Example fix

# before
text = hf_tok.decode(ids)  # NotImplementedError: tokenizer not loaded

# after
if not hf_tok._use_fallback():
    text = hf_tok.decode(ids)
else:
    raise RuntimeError("warm HF cache before running decode path")

Defensive patterns

Strategy: fallback

Validate before calling

tok = HuggingFaceTokenizer(model)
if tok._use_fallback():
    raise RuntimeError("HF tokenizer not loaded; warm cache or fix network before decode")

Type guard

def hf_decode_ready(tok) -> bool:
    return not tok._use_fallback()

Try / catch

try:
    text = tok.decode(ids)
except NotImplementedError as e:
    logger.warning("decode unavailable (fallback mode): %s", e)
    text = ""

Prevention

Warm the HF cache in a networked setup step.
Check _use_fallback() before decode-dependent logic.
Monitor tokenizer load failures and alert rather than silently estimating.

When it happens

Trigger: Same fallback condition as encode: tokenizer assets missing or unloadable (offline environment, bad model id, hub auth failure), followed by a call to decode(tokens).

Common situations: Air-gapped or proxied CI where huggingface_hub cannot fetch tokenizer.json; expired/gated-model access (HF_TOKEN missing for gated repos); cache corruption in HF_HOME.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/acfc7e20a6f69195. Report an issue: GitHub.