headroomlabs-ai/headroom · warning · NotImplementedError

{self.__class__.__name__} does not support encoding

Error message

{self.__class__.__name__} does not support encoding

What it means

BaseTokenizer.encode() is an optional capability: the base class default raises NotImplementedError naming the subclass. Counting tokens (the core API) is always available, but round-tripping text to token IDs is only implemented by backends with real vocabularies (tiktoken, HuggingFace, etc.).

Source

Thrown at headroom/tokenizers/base.py:441

        total += self.count_text(coerce_countable_text(function_call.get("arguments")))
        return total

    def encode(self, text: str) -> list[int]:
        """Encode text to token IDs.

        Optional method - not all backends support encoding.
        Default implementation raises NotImplementedError.

        Args:
            text: Text to encode.

        Returns:
            List of token IDs.

        Raises:
            NotImplementedError: If encoding is not supported.
        """
        raise NotImplementedError(f"{self.__class__.__name__} does not support encoding")

    def decode(self, tokens: list[int]) -> str:
        """Decode token IDs to text.

        Optional method - not all backends support decoding.
        Default implementation raises NotImplementedError.

        Args:
            tokens: List of token IDs.

        Returns:
            Decoded text.

        Raises:
            NotImplementedError: If decoding is not supported.
        """
        raise NotImplementedError(f"{self.__class__.__name__} does not support decoding")

View on GitHub (pinned to 322425c43b)

Solutions

1. Install a real backend (pip install tiktoken, or transformers for HuggingFace) so the registry resolves to an encoding-capable tokenizer.
2. Use a supported model name/tokenizer_name so backend detection picks tiktoken/HF instead of the estimator.
3. Only use count()/count_message_tokens() APIs if you do not need token IDs.
4. Check capability before calling: has_tokenizer() / is_available() classmethods where exposed.

Example fix

# before
tok = tokenizer_registry.get("internal-model-x")
ids = tok.encode("hello")  # NotImplementedError

# after
from headroom.tokenizers.tiktoken_counter import TiktokenTokenCounter
tok = TiktokenTokenCounter(encoding_name="cl100k_base")
ids = tok.encode("hello")

encodable = hasattr(tokenizer, "encode") and type(tokenizer).encode is not BaseTokenizer.encode

def supports_encoding(t) -> bool:
    try:
        t.encode("")
        return True
    except NotImplementedError:
        return False

try:
    ids = tokenizer.encode(text)
except NotImplementedError:
    ids = None  # estimation-only backend; rely on count()

When it happens

Trigger: Calling encode() on an estimating/heuristic tokenizer (e.g. EstimatingTokenCounter or a character-approximation backend) selected via the registry when no real backend is installed or the model is unknown.

Common situations: Using headroom in an environment without tiktoken/transformers installed; obscure or internal model names that fall through backend detection to the estimator; code that assumed encode() exists because it worked with another backend.

Related errors

• {self.__class__.__name__} does not support decoding
• Encoding not available for {self.model} - tokenizer {self.to
• Decoding not available for {self.model} - tokenizer {self.to
• mistral-common is required for MistralTokenizer. Install wit
• No tokenizer available for {model}: {e}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/8dad6de173c4d4ff. Report an issue: GitHub.