{"record":{"id":"f631638d651cf772","repo":"zylon-ai/private-gpt","slug":"unsupported-tokenizer-type-self-tokenizer","errorCode":null,"errorMessage":"Unsupported tokenizer: {type(self.tokenizer)}","messagePattern":"Unsupported tokenizer: (.+?)","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/mistral.py","lineNumber":363,"sourceCode":"        self.instruct = tokenizer.instruct_tokenizer\n        self.tokenizer = self.instruct.tokenizer\n\n        # Ensure test mode for proper validation\n        mode = tokenizer._chat_completion_request_validator._mode\n        if mode != ValidationMode.test:\n            raise ValueError(\n                \"Mistral tokenizer must be in test mode. Set \"\n                \"`mode=ValidationMode.test` when creating the tokenizer.\"\n            )\n\n        _mistral_version_str = str(self.tokenizer.version.value)\n        self.version: int = int(_mistral_version_str.split(\"v\")[-1])\n\n        self.is_tekken = isinstance(self.tokenizer, Tekkenizer)\n        self.is_spm = isinstance(self.tokenizer, SentencePieceTokenizer)\n\n        if not (self.is_tekken or self.is_spm):\n            raise TypeError(f\"Unsupported tokenizer: {type(self.tokenizer)}\")\n\n        # Build vocabulary dict (reverse order to keep lowest token id)\n        self._vocab = self.tokenizer.vocab()\n        self._max_token_id = self.vocab_size - 1\n\n        self._vocab_dict = {\n            self.convert_ids_to_tokens([i], skip_special_tokens=False)[0]: i\n            for i in range(self.vocab_size - 1, -1, -1)\n        }\n        self._vocab_dict = dict(sorted(self._vocab_dict.items(), key=lambda x: x[1]))\n\n        # Cache special tokens for performance\n        self._special_token_ids = self._get_special_token_ids()\n        self._special_token_ids_set = set(self._special_token_ids)\n        self._special_tokens = self._get_special_tokens(self._special_token_ids)\n        self._special_tokens_set = set(self._special_tokens)\n\n    def _get_special_token_ids(self) -> list[int]:","sourceCodeStart":345,"sourceCodeEnd":381,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/mistral.py#L345-L381","documentation":"After enforcing test mode, the constructor checks that the wrapped mistral_common tokenizer is a Tekkenizer or a SentencePieceTokenizer — the only two formats the wrapper implements (version parsing, vocab, token conversion all assume one of them). Any other tokenizer type raises TypeError.","triggerScenarios":"Wrapping a tokenizer from a newer/other mistral-common class (e.g. a future multimodal or HfTokenizer-based class) in MistralTokenizer. Not reachable via from_pretrained on standard repos, which yield tekken or SPM files.","commonSituations":"mistral-common version drift introducing a new tokenizer class; custom tokenizers passed into the wrapper; monkeypatched/mocked tokenizers in tests that are neither type.","solutions":["Pin/align mistral-common with the version the wrapper supports (the llm-mistral extra's pinned version).","Pass a real Tekkenizer or SentencePieceTokenizer instance (load one via MistralTokenizer.from_pretrained).","For tests, mock with one of the supported classes or patch the type checks rather than passing a bare fake."],"exampleFix":"# before\ntok = MistralTokenizer(fake_tokenizer)  # neither Tekkenizer nor SentencePieceTokenizer\n\n# after\ntok = MistralTokenizer.from_pretrained('mistralai/Mistral-Small-2412')","handlingStrategy":"type-guard","validationCode":"def is_supported_mistral_tokenizer(tok) -> bool:\n    from mistral_common.tokens.tokenizers.tekken import Tekkenizer\n    from mistral_common.tokens.tokenizers.sentencepiece import SentencePieceTokenizer\n    return isinstance(tok, (Tekkenizer, SentencePieceTokenizer))","typeGuard":"def is_supported_mistral_tokenizer(tok: object) -> bool:\n    tekken = getattr(type(tok), '__name__', '') == 'Tekkenizer'\n    spm = getattr(type(tok), '__name__', '') == 'SentencePieceTokenizer'\n    return tekken or spm","tryCatchPattern":"try:\n    wrapper = MistralTokenizer(raw)\nexcept TypeError as e:\n    if 'Unsupported tokenizer' in str(e):\n        raise RuntimeError('mistral-common version incompatible; pin the supported release') from e\n    raise","preventionTips":["Pin mistral-common to the project-validated version.","Use from_pretrained for all production loads.","When mocking in tests, subclass Tekkenizer/SentencePieceTokenizer or skip the wrapper."],"tags":["mistral","tokenizer","type-error","versioning"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}