{"record":{"id":"24b990fd2855577f","repo":"zylon-ai/private-gpt","slug":"mistraltokenizer-is-not-available-with-the-given-c","errorCode":null,"errorMessage":"MistralTokenizer is not available with the given configuration.","messagePattern":"MistralTokenizer is not available with the given configuration\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/registry.py","lineNumber":25,"sourceCode":"from private_gpt.components.llm.tokenizers.tokenizer_base import TokenizerBase\n\nTokenizerProvider = Callable[..., TokenizerBase]\n\n_EXTERNAL_TOKENIZER_FACTORIES: dict[str, TokenizerProvider] = {}\n\n\ndef register_tokenizer_factory(\n    tokenizer_mode: str,\n    factory: TokenizerProvider,\n) -> None:\n    _EXTERNAL_TOKENIZER_FACTORIES[tokenizer_mode] = factory\n\n\ndef _build_mistral_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.mistral import MistralTokenizer\n\n    if not MistralTokenizer.is_available(**kwargs):\n        raise ValueError(\n            \"MistralTokenizer is not available with the given configuration.\"\n        )\n\n    return MistralTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_tiktoken_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.tiktoken import TikTokenTokenizer\n\n    return TikTokenTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_estimator_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.estimator import EstimatorTokenizer\n\n    return EstimatorTokenizer.from_pretrained(**kwargs)\n\n","sourceCodeStart":7,"sourceCodeEnd":43,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/registry.py#L7-L43","documentation":"The 'mistral' entry in the tokenizer registry calls MistralTokenizer.is_available(**kwargs) first; that check simply requires a non-None model_id. If it is missing, the registry raises ValueError before attempting to load, telling you the mistral tokenizer cannot be built with the given configuration.","triggerScenarios":"TokenizerRegistry.get_tokenizer('mistral', ...) without a model_id kwarg (or model_id=None) — e.g. a component resolving the tokenizer from settings where llm.model was never set, or a remote-mode config that carries no model path.","commonSituations":"Switching tokenizer_mode to 'mistral' while the LLM settings only define an API base/key and no local/Hub model; optional-model DI wiring where model is None for remote backends; env-specific settings files omitting the model key.","solutions":["Pass a model_id when requesting the mistral tokenizer: TokenizerRegistry.get_tokenizer('mistral', model_id='mistralai/Mistral-Small-2412').","Ensure the settings feeding the tokenizer (e.g. llm model field) are populated in every environment that selects tokenizer_mode='mistral'.","If you have no model id, use a mode that does not need one ('estimator') or rely on 'default', which falls back automatically."],"exampleFix":"# before\ntok = TokenizerRegistry.get_tokenizer('mistral', model_id=settings.model)  # settings.model is None\n\n# after\nassert settings.model, 'tokenizer_mode=mistral requires a model id'\ntok = TokenizerRegistry.get_tokenizer('mistral', model_id=settings.model)","handlingStrategy":"validation","validationCode":"def mistral_mode_config_ok(settings) -> bool:\n    if getattr(settings, 'tokenizer_mode', None) != 'mistral':\n        return True\n    return bool(getattr(settings, 'model', None))\n\nassert mistral_mode_config_ok(settings), 'tokenizer_mode=mistral requires a model id'","typeGuard":null,"tryCatchPattern":"try:\n    tok = TokenizerRegistry.get_tokenizer('mistral', model_id=model_id)\nexcept ValueError as e:\n    if 'not available with the given configuration' in str(e):\n        raise ConfigurationError('mistral tokenizer needs model_id') from e\n    raise","preventionTips":["Set a model id whenever tokenizer_mode='mistral' is enabled.","Validate tokenizer settings per environment at startup.","Use 'default' mode in remote-only deployments so the registry can fall back gracefully."],"tags":["configuration","mistral","tokenizer","registry"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}