{"record":{"id":"4442819ef65ae26e","repo":"headroomlabs-ai/headroom","slug":"mistral-common-is-required-for-mistraltokenizer-i","errorCode":null,"errorMessage":"mistral-common is required for MistralTokenizer. Install with: pip install mistral-common","messagePattern":"mistral-common is required for MistralTokenizer\\. Install with: pip install mistral-common","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"headroom/tokenizers/mistral.py","lineNumber":69,"sourceCode":"    \"codestral\": \"v3\",\n    \"codestral-latest\": \"v3\",\n    # Mixtral uses v1\n    \"mixtral-8x7b\": \"v1\",\n    \"mixtral-8x22b\": \"v1\",\n    \"open-mixtral-8x7b\": \"v1\",\n    \"open-mixtral-8x22b\": \"v1\",\n    # Mistral 7B uses v1\n    \"mistral-7b\": \"v1\",\n    \"open-mistral-7b\": \"v1\",\n    \"mistral-7b-instruct\": \"v1\",\n}\n\n\n@lru_cache(maxsize=4)\ndef _get_tokenizer(version: str):\n    \"\"\"Get and cache Mistral tokenizer by version.\"\"\"\n    if not MISTRAL_AVAILABLE:\n        raise RuntimeError(\n            \"mistral-common is required for MistralTokenizer. \"\n            \"Install with: pip install mistral-common\"\n        )\n\n    if version == \"v3\":\n        return _MistralTokenizer.v3(is_tekken=True)\n    elif version == \"v2\":\n        return _MistralTokenizer.v2()\n    else:  # v1\n        return _MistralTokenizer.v1()\n\n\ndef get_tokenizer_version(model: str) -> str:\n    \"\"\"Get tokenizer version for a model.\"\"\"\n    model_lower = model.lower()\n\n    # Direct lookup\n    if model_lower in MODEL_TO_VERSION:","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/tokenizers/mistral.py#L51-L87","documentation":"The cached factory _get_tokenizer(version) raises RuntimeError when the optional mistral-common package is not installed. mistral-common ships the official MistralTokenizer (v1/v2/v3/tekken) implementations, so without it no Mistral tokenizer can be constructed. The error includes the exact pip install line.","triggerScenarios":"Registry resolves a mistral-* model to the mistral backend while headroom was installed without the mistral extra; calling MistralTokenizer(...).tokenizer or get_tokenizer_version-driven tokenization in a minimal environment.","commonSituations":"Base `pip install headroom` without extras; CI images trimmed of optional deps; deploying the proxy where only tiktoken was expected but a request names a mistral model.","solutions":["pip install mistral-common (or headroom's mistral extra if provided, e.g. headroom[mistral]).","Verify with headroom.tokenizers.mistral.is_mistral_available() before routing mistral models.","If mistral support is unneeded, ensure the registry falls back to estimation (fallback=True) rather than the mistral backend."],"exampleFix":"# before\ntok = MistralTokenizer(\"mistral-large\")  # RuntimeError: mistral-common required\n\n# after\n# shell: pip install mistral-common\nfrom headroom.tokenizers.mistral import is_mistral_available\nassert is_mistral_available(), \"pip install mistral-common\"\ntok = MistralTokenizer(\"mistral-large\")","handlingStrategy":"validation","validationCode":"from headroom.tokenizers.mistral import is_mistral_available\nif not is_mistral_available():\n    raise RuntimeError(\"pip install mistral-common\")","typeGuard":"def mistral_supported() -> bool:\n    from headroom.tokenizers.mistral import is_mistral_available\n    return is_mistral_available()","tryCatchPattern":"try:\n    tok = MistralTokenizer(model)\nexcept RuntimeError as e:\n    if \"mistral-common\" in str(e):\n        tok = EstimatingTokenCounter()  # documented degraded mode\n    else:\n        raise","preventionTips":["Declare mistral-common in deps if mistral models are in scope.","Gate mistral model routing on is_mistral_available().","Add a startup dependency check for optional tokenizer extras."],"tags":["tokenizer","mistral","dependencies","optional-import"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}