zylon-ai/private-gpt · error · TypeError

Unsupported tokenizer: {type(self.tokenizer)}

Error message

Unsupported tokenizer: {type(self.tokenizer)}

What it means

After enforcing test mode, the constructor checks that the wrapped mistral_common tokenizer is a Tekkenizer or a SentencePieceTokenizer — the only two formats the wrapper implements (version parsing, vocab, token conversion all assume one of them). Any other tokenizer type raises TypeError.

Source

Thrown at private_gpt/components/llm/tokenizers/mistral.py:363

        self.instruct = tokenizer.instruct_tokenizer
        self.tokenizer = self.instruct.tokenizer

        # Ensure test mode for proper validation
        mode = tokenizer._chat_completion_request_validator._mode
        if mode != ValidationMode.test:
            raise ValueError(
                "Mistral tokenizer must be in test mode. Set "
                "`mode=ValidationMode.test` when creating the tokenizer."
            )

        _mistral_version_str = str(self.tokenizer.version.value)
        self.version: int = int(_mistral_version_str.split("v")[-1])

        self.is_tekken = isinstance(self.tokenizer, Tekkenizer)
        self.is_spm = isinstance(self.tokenizer, SentencePieceTokenizer)

        if not (self.is_tekken or self.is_spm):
            raise TypeError(f"Unsupported tokenizer: {type(self.tokenizer)}")

        # Build vocabulary dict (reverse order to keep lowest token id)
        self._vocab = self.tokenizer.vocab()
        self._max_token_id = self.vocab_size - 1

        self._vocab_dict = {
            self.convert_ids_to_tokens([i], skip_special_tokens=False)[0]: i
            for i in range(self.vocab_size - 1, -1, -1)
        }
        self._vocab_dict = dict(sorted(self._vocab_dict.items(), key=lambda x: x[1]))

        # Cache special tokens for performance
        self._special_token_ids = self._get_special_token_ids()
        self._special_token_ids_set = set(self._special_token_ids)
        self._special_tokens = self._get_special_tokens(self._special_token_ids)
        self._special_tokens_set = set(self._special_tokens)

    def _get_special_token_ids(self) -> list[int]:

View on GitHub (pinned to 4a030776a3)

Solutions

1. Pin/align mistral-common with the version the wrapper supports (the llm-mistral extra's pinned version).
2. Pass a real Tekkenizer or SentencePieceTokenizer instance (load one via MistralTokenizer.from_pretrained).
3. For tests, mock with one of the supported classes or patch the type checks rather than passing a bare fake.

Example fix

# before
tok = MistralTokenizer(fake_tokenizer)  # neither Tekkenizer nor SentencePieceTokenizer

# after
tok = MistralTokenizer.from_pretrained('mistralai/Mistral-Small-2412')

def is_supported_mistral_tokenizer(tok) -> bool:
    from mistral_common.tokens.tokenizers.tekken import Tekkenizer
    from mistral_common.tokens.tokenizers.sentencepiece import SentencePieceTokenizer
    return isinstance(tok, (Tekkenizer, SentencePieceTokenizer))

def is_supported_mistral_tokenizer(tok: object) -> bool:
    tekken = getattr(type(tok), '__name__', '') == 'Tekkenizer'
    spm = getattr(type(tok), '__name__', '') == 'SentencePieceTokenizer'
    return tekken or spm

try:
    wrapper = MistralTokenizer(raw)
except TypeError as e:
    if 'Unsupported tokenizer' in str(e):
        raise RuntimeError('mistral-common version incompatible; pin the supported release') from e
    raise

When it happens

Trigger: Wrapping a tokenizer from a newer/other mistral-common class (e.g. a future multimodal or HfTokenizer-based class) in MistralTokenizer. Not reachable via from_pretrained on standard repos, which yield tekken or SPM files.

Common situations: mistral-common version drift introducing a new tokenizer class; custom tokenizers passed into the wrapper; monkeypatched/mocked tokenizers in tests that are neither type.

Related errors

• Mistral tokenizer dependencies are not installed. Install wi
• Found {len(matched_files)} files matching the pattern: {file
• Found {len(matched_files)} files matching the pattern: {file
• Mistral tokenizer must be in test mode. Set `mode=Validation
• Empty response from Mistral tokenizer

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/f631638d651cf772. Report an issue: GitHub.