{"record":{"id":"bf20c263b1c3b4c2","repo":"zylon-ai/private-gpt","slug":"mistral-tokenizer-must-be-in-test-mode-set-mode","errorCode":null,"errorMessage":"Mistral tokenizer must be in test mode. Set `mode=ValidationMode.test` when creating the tokenizer.","messagePattern":"Mistral tokenizer must be in test mode\\. Set `mode=ValidationMode\\.test` when creating the tokenizer\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/mistral.py","lineNumber":351,"sourceCode":"    ) -> None:\n        ValidationMode = _load_mistral_module(\n            \"mistral_common.protocol.instruct.validator\"\n        ).ValidationMode\n        SentencePieceTokenizer = _load_mistral_module(\n            \"mistral_common.tokens.tokenizers.sentencepiece\"\n        ).SentencePieceTokenizer\n        Tekkenizer = _load_mistral_module(\n            \"mistral_common.tokens.tokenizers.tekken\"\n        ).Tekkenizer\n\n        self.mistral = tokenizer\n        self.instruct = tokenizer.instruct_tokenizer\n        self.tokenizer = self.instruct.tokenizer\n\n        # Ensure test mode for proper validation\n        mode = tokenizer._chat_completion_request_validator._mode\n        if mode != ValidationMode.test:\n            raise ValueError(\n                \"Mistral tokenizer must be in test mode. Set \"\n                \"`mode=ValidationMode.test` when creating the tokenizer.\"\n            )\n\n        _mistral_version_str = str(self.tokenizer.version.value)\n        self.version: int = int(_mistral_version_str.split(\"v\")[-1])\n\n        self.is_tekken = isinstance(self.tokenizer, Tekkenizer)\n        self.is_spm = isinstance(self.tokenizer, SentencePieceTokenizer)\n\n        if not (self.is_tekken or self.is_spm):\n            raise TypeError(f\"Unsupported tokenizer: {type(self.tokenizer)}\")\n\n        # Build vocabulary dict (reverse order to keep lowest token id)\n        self._vocab = self.tokenizer.vocab()\n        self._max_token_id = self.vocab_size - 1\n\n        self._vocab_dict = {","sourceCodeStart":333,"sourceCodeEnd":369,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/mistral.py#L333-L369","documentation":"MistralTokenizer's constructor reads tokenizer._chat_completion_request_validator._mode and requires ValidationMode.test, because encode_chat_completion performs request validation that only behaves predictably (e.g. tolerates test-mode tool-call ids) in test mode. from_pretrained always constructs the underlying tokenizer with mode=ValidationMode.test; passing one built otherwise raises ValueError.","triggerScenarios":"Constructing MistralTokenizer directly with a mistral_common PublicMistralTokenizer created via MistralTokenizer.from_file(path, mode=ValidationMode.normal) or another mode, instead of going through MistralTokenizer.from_pretrained.","commonSituations":"Custom code wrapping an existing mistral_common tokenizer; upgrading mistral-common where default modes changed; tests instantiating the wrapper by hand.","solutions":["Load via MistralTokenizer.from_pretrained(model_id), which sets mode=ValidationMode.test for you.","If constructing manually, create the underlying tokenizer with PublicMistralTokenizer.from_file(file, mode=ValidationMode.test).","Check the mode before wrapping and rebuild the tokenizer if it is not test."],"exampleFix":"# before\nraw = PublicMistralTokenizer.from_file('tekken.json', mode=ValidationMode.normal)\ntok = MistralTokenizer(raw)\n\n# after\nraw = PublicMistralTokenizer.from_file('tekken.json', mode=ValidationMode.test)\ntok = MistralTokenizer(raw)","handlingStrategy":"validation","validationCode":"def is_test_mode(mistral_tokenizer) -> bool:\n    mode = mistral_tokenizer._chat_completion_request_validator._mode\n    from mistral_common.protocol.instruct.validator import ValidationMode\n    return mode == ValidationMode.test\n\nassert is_test_mode(raw_tok)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always build via MistralTokenizer.from_pretrained rather than wrapping raw tokenizers.","If wrapping manually, pass mode=ValidationMode.test to PublicMistralTokenizer.from_file.","Add a construction smoke test for any custom tokenizer wiring."],"tags":["mistral","tokenizer","validation-mode","api-misuse"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}