{"record":{"id":"25cfb6b2d7e07616","repo":"headroomlabs-ai/headroom","slug":"no-tokenizer-available-for-model-e","errorCode":null,"errorMessage":"No tokenizer available for {model}: {e}","messagePattern":"No tokenizer available for (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/tokenizers/registry.py","lineNumber":213,"sourceCode":"        # Check cache\n        cache_key = f\"{model_lower}:{backend or 'auto'}\"\n        if cache_key in registry._cache:\n            return registry._cache[cache_key]\n\n        # Create tokenizer\n        try:\n            tokenizer = registry._create_tokenizer(model, backend)\n            registry._cache[cache_key] = tokenizer\n            return tokenizer\n        except Exception as e:\n            if fallback:\n                logger.warning(\n                    f\"Failed to create tokenizer for {model}: {e}. Falling back to estimation.\"\n                )\n                tokenizer = EstimatingTokenCounter()\n                registry._cache[cache_key] = tokenizer\n                return tokenizer\n            raise ValueError(f\"No tokenizer available for {model}: {e}\") from e\n\n    @classmethod\n    def register(\n        cls,\n        model: str,\n        tokenizer: TokenCounter | None = None,\n        factory: Callable[[str], TokenCounter] | None = None,\n    ) -> None:\n        \"\"\"Register a tokenizer or factory for a model.\n\n        Args:\n            model: Model name to register.\n            tokenizer: Pre-instantiated tokenizer instance.\n            factory: Factory function that creates tokenizer for model.\n\n        Raises:\n            ValueError: If neither tokenizer nor factory provided.\n        \"\"\"","sourceCodeStart":195,"sourceCodeEnd":231,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/tokenizers/registry.py#L195-L231","documentation":"TokenizerRegistry.get() wraps any exception raised while creating a tokenizer for a model; when fallback is disabled it re-raises as ValueError('No tokenizer available for {model}: {e}') chaining the original error. This is the terminal failure of backend resolution: auto-detect picked (or you passed) a backend and its factory threw.","triggerScenarios":"get(model, backend='huggingface', fallback=False) where transformers is missing; a mistral model routed to the mistral backend without mistral-common; a factory registered via register() that throws; tiktoken load failures with fallback disabled.","commonSituations":"Explicitly disabling fallback to force exact counting in billing-sensitive code; prod images missing optional tokenizer deps; models whose detected backend depends on an uninstalled package.","solutions":["Read the chained exception (`raise ... from e` — inspect __cause__) to see which backend failed and why, then install that dependency (tiktoken / transformers / mistral-common) or fix the factory.","Allow fallback=True (or omit the flag) so the registry degrades to EstimatingTokenCounter with a warning instead of raising.","Register a working factory for the model via TokenizerRegistry.register(model, factory=...) before calling get()."],"exampleFix":"# before\ntok = TokenizerRegistry.get(\"mistral-large\", fallback=False)  # ValueError\n\n# after\ntry:\n    tok = TokenizerRegistry.get(\"mistral-large\", fallback=False)\nexcept ValueError as e:\n    logger.error(\"backend failed: %s\", e.__cause__)\n    tok = TokenizerRegistry.get(\"mistral-large\", fallback=True)","handlingStrategy":"fallback","validationCode":"try:\n    TokenizerRegistry.get(model, backend=backend, fallback=False)\nexcept ValueError:\n    ok = False  # decide fallback policy before the real call","typeGuard":null,"tryCatchPattern":"try:\n    tok = TokenizerRegistry.get(model, fallback=False)\nexcept ValueError as e:\n    logger.error(\"tokenizer backend failed for %s: %s\", model, e.__cause__)\n    tok = TokenizerRegistry.get(model, fallback=True)","preventionTips":["Default to fallback=True in request paths; reserve fallback=False for startup assertions.","Install all tokenizer extras your model mix needs.","Log the chained cause (__cause__) to identify the failing backend."],"tags":["tokenizer","registry","fallback","dependencies"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}