headroomlabs-ai/headroom · error · ValueError

No tokenizer available for {model}: {e}

Error message

No tokenizer available for {model}: {e}

What it means

TokenizerRegistry.get() wraps any exception raised while creating a tokenizer for a model; when fallback is disabled it re-raises as ValueError('No tokenizer available for {model}: {e}') chaining the original error. This is the terminal failure of backend resolution: auto-detect picked (or you passed) a backend and its factory threw.

Source

Thrown at headroom/tokenizers/registry.py:213

        # Check cache
        cache_key = f"{model_lower}:{backend or 'auto'}"
        if cache_key in registry._cache:
            return registry._cache[cache_key]

        # Create tokenizer
        try:
            tokenizer = registry._create_tokenizer(model, backend)
            registry._cache[cache_key] = tokenizer
            return tokenizer
        except Exception as e:
            if fallback:
                logger.warning(
                    f"Failed to create tokenizer for {model}: {e}. Falling back to estimation."
                )
                tokenizer = EstimatingTokenCounter()
                registry._cache[cache_key] = tokenizer
                return tokenizer
            raise ValueError(f"No tokenizer available for {model}: {e}") from e

    @classmethod
    def register(
        cls,
        model: str,
        tokenizer: TokenCounter | None = None,
        factory: Callable[[str], TokenCounter] | None = None,
    ) -> None:
        """Register a tokenizer or factory for a model.

        Args:
            model: Model name to register.
            tokenizer: Pre-instantiated tokenizer instance.
            factory: Factory function that creates tokenizer for model.

        Raises:
            ValueError: If neither tokenizer nor factory provided.
        """

View on GitHub (pinned to 322425c43b)

Solutions

1. Read the chained exception (`raise ... from e` — inspect __cause__) to see which backend failed and why, then install that dependency (tiktoken / transformers / mistral-common) or fix the factory.
2. Allow fallback=True (or omit the flag) so the registry degrades to EstimatingTokenCounter with a warning instead of raising.
3. Register a working factory for the model via TokenizerRegistry.register(model, factory=...) before calling get().

Example fix

# before
tok = TokenizerRegistry.get("mistral-large", fallback=False)  # ValueError

# after
try:
    tok = TokenizerRegistry.get("mistral-large", fallback=False)
except ValueError as e:
    logger.error("backend failed: %s", e.__cause__)
    tok = TokenizerRegistry.get("mistral-large", fallback=True)

try:
    TokenizerRegistry.get(model, backend=backend, fallback=False)
except ValueError:
    ok = False  # decide fallback policy before the real call

try:
    tok = TokenizerRegistry.get(model, fallback=False)
except ValueError as e:
    logger.error("tokenizer backend failed for %s: %s", model, e.__cause__)
    tok = TokenizerRegistry.get(model, fallback=True)

When it happens

Trigger: get(model, backend='huggingface', fallback=False) where transformers is missing; a mistral model routed to the mistral backend without mistral-common; a factory registered via register() that throws; tiktoken load failures with fallback disabled.

Common situations: Explicitly disabling fallback to force exact counting in billing-sensitive code; prod images missing optional tokenizer deps; models whose detected backend depends on an uninstalled package.

Related errors

• {self.__class__.__name__} does not support encoding
• {self.__class__.__name__} does not support decoding
• Encoding not available for {self.model} - tokenizer {self.to
• Decoding not available for {self.model} - tokenizer {self.to
• mistral-common is required for MistralTokenizer. Install wit

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/25cfb6b2d7e07616. Report an issue: GitHub.