zylon-ai/private-gpt · error · ValueError

Mistral tokenizer must be in test mode. Set `mode=Validation

Error message

Mistral tokenizer must be in test mode. Set `mode=ValidationMode.test` when creating the tokenizer.

What it means

MistralTokenizer's constructor reads tokenizer._chat_completion_request_validator._mode and requires ValidationMode.test, because encode_chat_completion performs request validation that only behaves predictably (e.g. tolerates test-mode tool-call ids) in test mode. from_pretrained always constructs the underlying tokenizer with mode=ValidationMode.test; passing one built otherwise raises ValueError.

Source

Thrown at private_gpt/components/llm/tokenizers/mistral.py:351

    ) -> None:
        ValidationMode = _load_mistral_module(
            "mistral_common.protocol.instruct.validator"
        ).ValidationMode
        SentencePieceTokenizer = _load_mistral_module(
            "mistral_common.tokens.tokenizers.sentencepiece"
        ).SentencePieceTokenizer
        Tekkenizer = _load_mistral_module(
            "mistral_common.tokens.tokenizers.tekken"
        ).Tekkenizer

        self.mistral = tokenizer
        self.instruct = tokenizer.instruct_tokenizer
        self.tokenizer = self.instruct.tokenizer

        # Ensure test mode for proper validation
        mode = tokenizer._chat_completion_request_validator._mode
        if mode != ValidationMode.test:
            raise ValueError(
                "Mistral tokenizer must be in test mode. Set "
                "`mode=ValidationMode.test` when creating the tokenizer."
            )

        _mistral_version_str = str(self.tokenizer.version.value)
        self.version: int = int(_mistral_version_str.split("v")[-1])

        self.is_tekken = isinstance(self.tokenizer, Tekkenizer)
        self.is_spm = isinstance(self.tokenizer, SentencePieceTokenizer)

        if not (self.is_tekken or self.is_spm):
            raise TypeError(f"Unsupported tokenizer: {type(self.tokenizer)}")

        # Build vocabulary dict (reverse order to keep lowest token id)
        self._vocab = self.tokenizer.vocab()
        self._max_token_id = self.vocab_size - 1

        self._vocab_dict = {

View on GitHub (pinned to 4a030776a3)

Solutions

1. Load via MistralTokenizer.from_pretrained(model_id), which sets mode=ValidationMode.test for you.
2. If constructing manually, create the underlying tokenizer with PublicMistralTokenizer.from_file(file, mode=ValidationMode.test).
3. Check the mode before wrapping and rebuild the tokenizer if it is not test.

Example fix

# before
raw = PublicMistralTokenizer.from_file('tekken.json', mode=ValidationMode.normal)
tok = MistralTokenizer(raw)

# after
raw = PublicMistralTokenizer.from_file('tekken.json', mode=ValidationMode.test)
tok = MistralTokenizer(raw)

def is_test_mode(mistral_tokenizer) -> bool:
    mode = mistral_tokenizer._chat_completion_request_validator._mode
    from mistral_common.protocol.instruct.validator import ValidationMode
    return mode == ValidationMode.test

assert is_test_mode(raw_tok)

When it happens

Trigger: Constructing MistralTokenizer directly with a mistral_common PublicMistralTokenizer created via MistralTokenizer.from_file(path, mode=ValidationMode.normal) or another mode, instead of going through MistralTokenizer.from_pretrained.

Common situations: Custom code wrapping an existing mistral_common tokenizer; upgrading mistral-common where default modes changed; tests instantiating the wrapper by hand.

Related errors

• Mistral tokenizer dependencies are not installed. Install wi
• Found {len(matched_files)} files matching the pattern: {file
• Found {len(matched_files)} files matching the pattern: {file
• Cannot set both `add_generation_prompt` and `continue_final_
• Unsupported tokenizer: {type(self.tokenizer)}

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/bf20c263b1c3b4c2. Report an issue: GitHub.