{"record":{"id":"becba1b2be78bf45","repo":"zylon-ai/private-gpt","slug":"found-len-matched-files-files-matching-the-patt","errorCode":null,"errorMessage":"Found {len(matched_files)} files matching the pattern: {file_pattern}. Make sure only one Mistral tokenizer is present in {files}.","messagePattern":"Found (.+?) files matching the pattern: (.+?)\\. Make sure only one Mistral tokenizer is present in (.+?)\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/mistral.py","lineNumber":190,"sourceCode":"                revision = file.read()\n\n    if revision:\n        revision_dir = os.path.join(repo_cache, \"snapshots\", revision)\n        if os.path.isdir(revision_dir):\n            return os.listdir(revision_dir)\n\n    return []\n\n\ndef find_tokenizer_file(files: list[str]) -> str:\n    \"\"\"Find the Mistral tokenizer file from a list of repository files.\"\"\"\n    file_pattern = re.compile(\n        r\"^tokenizer\\.model\\.v.*$|^tekken\\.json$|^tokenizer\\.mm\\.model\\.v.*$\"\n    )\n\n    matched_files = [file for file in files if file_pattern.match(file)]\n    if len(matched_files) > 1:\n        raise OSError(\n            f\"Found {len(matched_files)} files matching the \"\n            f\"pattern: {file_pattern}. Make sure only one Mistral \"\n            f\"tokenizer is present in {files}.\"\n        )\n    elif len(matched_files) == 0:\n        raise OSError(\n            f\"Found {len(matched_files)} files matching the \"\n            f\"pattern: {file_pattern}. Make sure that a Mistral \"\n            f\"tokenizer is present in {files}.\"\n        )\n\n    return matched_files[0]\n\n\ndef _prepare_apply_chat_template_tools_and_messages(\n    messages: list[dict[str, Any]],\n    tools: list[dict[str, Any]] | None = None,\n    continue_final_message: bool = False,","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/mistral.py#L172-L208","documentation":"find_tokenizer_file scans a local model directory for Mistral tokenizer files matching tokenizer.model.v*, tekken.json, or tokenizer.mm.model.v*. Finding more than one match is ambiguous (the loader would not know which tokenizer to use), so it raises OSError demanding exactly one.","triggerScenarios":"MistralTokenizer.from_pretrained(<local dir>) where the directory contains e.g. both tokenizer.model.v3 and tekken.json — common when someone merged files from two checkpoints or copied a v*-style and a Tekken tokenizer into one folder.","commonSituations":"Manual model packaging; converting a repo between SPM and Tekken formats without cleaning up; fine-tune artifacts that inherited extra tokenizer files from the base model.","solutions":["Inspect the directory: ls <model_dir> and identify the duplicates matched by the pattern.","Keep only the tokenizer file matching the model's actual format (tekken.json for Tekken models, tokenizer.model.vN for SPM v3/v7) and delete/move the others.","Alternatively point model_id at a clean directory containing a single tokenizer file."],"exampleFix":"# before: model_dir contains tokenizer.model.v3 AND tekken.json\n# after: keep only the correct one\nrm model_dir/tokenizer.model.v3   # model is Tekken-based\ntok = MistralTokenizer.from_pretrained('model_dir')","handlingStrategy":"validation","validationCode":"import re\n Mistral_RE = re.compile(r'^tokenizer\\.model\\.v.*$|^tekken\\.json$|^tokenizer\\.mm\\.model\\.v.*$')\n\ndef dir_has_single_mistral_tokenizer(path: str) -> bool:\n    import os\n    matches = [f for f in os.listdir(path) if Mistral_RE.match(f)]\n    return len(matches) == 1","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Package local model dirs with exactly one tokenizer artifact.","Validate downloaded model directories before registering them as local models.","When converting between SPM and Tekken formats, delete the superseded tokenizer file."],"tags":["mistral","tokenizer","local-model","filesystem"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}