headroomlabs-ai/headroom · error · RuntimeError

mistral-common is required for MistralTokenizer. Install wit

Error message

mistral-common is required for MistralTokenizer. Install with: pip install mistral-common

What it means

The cached factory _get_tokenizer(version) raises RuntimeError when the optional mistral-common package is not installed. mistral-common ships the official MistralTokenizer (v1/v2/v3/tekken) implementations, so without it no Mistral tokenizer can be constructed. The error includes the exact pip install line.

Source

Thrown at headroom/tokenizers/mistral.py:69

    "codestral": "v3",
    "codestral-latest": "v3",
    # Mixtral uses v1
    "mixtral-8x7b": "v1",
    "mixtral-8x22b": "v1",
    "open-mixtral-8x7b": "v1",
    "open-mixtral-8x22b": "v1",
    # Mistral 7B uses v1
    "mistral-7b": "v1",
    "open-mistral-7b": "v1",
    "mistral-7b-instruct": "v1",
}


@lru_cache(maxsize=4)
def _get_tokenizer(version: str):
    """Get and cache Mistral tokenizer by version."""
    if not MISTRAL_AVAILABLE:
        raise RuntimeError(
            "mistral-common is required for MistralTokenizer. "
            "Install with: pip install mistral-common"
        )

    if version == "v3":
        return _MistralTokenizer.v3(is_tekken=True)
    elif version == "v2":
        return _MistralTokenizer.v2()
    else:  # v1
        return _MistralTokenizer.v1()


def get_tokenizer_version(model: str) -> str:
    """Get tokenizer version for a model."""
    model_lower = model.lower()

    # Direct lookup
    if model_lower in MODEL_TO_VERSION:

View on GitHub (pinned to 322425c43b)

Solutions

pip install mistral-common (or headroom's mistral extra if provided, e.g. headroom[mistral]).
Verify with headroom.tokenizers.mistral.is_mistral_available() before routing mistral models.
If mistral support is unneeded, ensure the registry falls back to estimation (fallback=True) rather than the mistral backend.

Example fix

# before
tok = MistralTokenizer("mistral-large")  # RuntimeError: mistral-common required

# after
# shell: pip install mistral-common
from headroom.tokenizers.mistral import is_mistral_available
assert is_mistral_available(), "pip install mistral-common"
tok = MistralTokenizer("mistral-large")

Defensive patterns

Strategy: validation

Validate before calling

from headroom.tokenizers.mistral import is_mistral_available
if not is_mistral_available():
    raise RuntimeError("pip install mistral-common")

Type guard

def mistral_supported() -> bool:
    from headroom.tokenizers.mistral import is_mistral_available
    return is_mistral_available()

Try / catch

try:
    tok = MistralTokenizer(model)
except RuntimeError as e:
    if "mistral-common" in str(e):
        tok = EstimatingTokenCounter()  # documented degraded mode
    else:
        raise

Prevention

Declare mistral-common in deps if mistral models are in scope.
Gate mistral model routing on is_mistral_available().
Add a startup dependency check for optional tokenizer extras.

When it happens

Trigger: Registry resolves a mistral-* model to the mistral backend while headroom was installed without the mistral extra; calling MistralTokenizer(...).tokenizer or get_tokenizer_version-driven tokenization in a minimal environment.

Common situations: Base `pip install headroom` without extras; CI images trimmed of optional deps; deploying the proxy where only tiktoken was expected but a request names a mistral model.

Related errors

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/4442819ef65ae26e. Report an issue: GitHub.