headroomlabs-ai/headroom · warning · NotImplementedError

Encoding not available for {self.model} - tokenizer {self.to

Error message

Encoding not available for {self.model} - tokenizer {self.tokenizer_name} could not be loaded

What it means

HuggingFaceTokenizer.encode() raises NotImplementedError when the underlying transformers tokenizer could not be loaded and the instance is running in fallback (count-only estimation) mode. The message names the model and the tokenizer that failed to load, distinguishing 'temporarily unavailable' from 'never supported'.

Source

Thrown at headroom/tokenizers/huggingface.py:365

                # Fall back to base implementation
                pass

        return super().count_messages(messages)

    def encode(self, text: str) -> list[int]:
        """Encode text to token IDs.

        Args:
            text: Text to encode.

        Returns:
            List of token IDs.

        Raises:
            NotImplementedError: If tokenizer not available.
        """
        if self._use_fallback():
            raise NotImplementedError(
                f"Encoding not available for {self.model} - "
                f"tokenizer {self.tokenizer_name} could not be loaded"
            )
        return self.tokenizer.encode(text, add_special_tokens=False)

    def decode(self, tokens: list[int]) -> str:
        """Decode token IDs to text.

        Args:
            tokens: List of token IDs.

        Returns:
            Decoded text.

        Raises:
            NotImplementedError: If tokenizer not available.
        """
        if self._use_fallback():

View on GitHub (pinned to 322425c43b)

Solutions

1. Fix the load: correct the model id, ensure network access / HF_TOKEN, or pre-download: `huggingface-cli download <model>` with HF_HOME set to a shared cache.
2. Run with HF_HUB_OFFLINE=1 only after warming the cache so is_available()/lazy load succeeds.
3. If encode is optional, catch NotImplementedError and degrade to count()-only behavior.

Example fix

# before
tok = HuggingFaceTokenizer("mistralai/Mistral-7B-v0.1")  # load failed -> fallback
ids = tok.encode(text)  # NotImplementedError

# after
# pre-warm in a networked step: huggingface-cli download mistralai/Mistral-7B-v0.1
tok = HuggingFaceTokenizer("mistralai/Mistral-7B-v0.1")
ids = tok.encode(text)

from headroom.tokenizers.huggingface import HuggingFaceTokenizer
assert HuggingFaceTokenizer.is_available(), "install transformers"
# probe load success without raising:
tok = HuggingFaceTokenizer(model)
assert not tok._use_fallback(), f"tokenizer {tok.tokenizer_name} failed to load"

def hf_ready(tok) -> bool:
    return not tok._use_fallback()

try:
    ids = tok.encode(text)
except NotImplementedError as e:
    logger.warning("HF tokenizer in fallback mode: %s", e)
    ids = None

When it happens

Trigger: Constructing HuggingFaceTokenizer for a model whose tokenizer files failed to download (offline env, HF hub blocked, bad model id) — _use_fallback() becomes true — then calling encode(); count() still works via estimation.

Common situations: CI without HF_TOKEN or with restricted egress to huggingface.co; typo'd or retired HF model ids; corporate proxies breaking huggingface_hub downloads; transient hub outages leaving no local cache.

Related errors

• Decoding not available for {self.model} - tokenizer {self.to
• {self.__class__.__name__} does not support encoding
• {self.__class__.__name__} does not support decoding
• No tokenizer available for {model}: {e}
• tiktoken encoding {encoding_name!r} previously failed to loa

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/7971256f01213d7f. Report an issue: GitHub.