zylon-ai/private-gpt · error · OSError

Found {len(matched_files)} files matching the pattern: {file

Error message

Found {len(matched_files)} files matching the pattern: {file_pattern}. Make sure only one Mistral tokenizer is present in {files}.

What it means

find_tokenizer_file scans a local model directory for Mistral tokenizer files matching tokenizer.model.v*, tekken.json, or tokenizer.mm.model.v*. Finding more than one match is ambiguous (the loader would not know which tokenizer to use), so it raises OSError demanding exactly one.

Source

Thrown at private_gpt/components/llm/tokenizers/mistral.py:190

                revision = file.read()

    if revision:
        revision_dir = os.path.join(repo_cache, "snapshots", revision)
        if os.path.isdir(revision_dir):
            return os.listdir(revision_dir)

    return []


def find_tokenizer_file(files: list[str]) -> str:
    """Find the Mistral tokenizer file from a list of repository files."""
    file_pattern = re.compile(
        r"^tokenizer\.model\.v.*$|^tekken\.json$|^tokenizer\.mm\.model\.v.*$"
    )

    matched_files = [file for file in files if file_pattern.match(file)]
    if len(matched_files) > 1:
        raise OSError(
            f"Found {len(matched_files)} files matching the "
            f"pattern: {file_pattern}. Make sure only one Mistral "
            f"tokenizer is present in {files}."
        )
    elif len(matched_files) == 0:
        raise OSError(
            f"Found {len(matched_files)} files matching the "
            f"pattern: {file_pattern}. Make sure that a Mistral "
            f"tokenizer is present in {files}."
        )

    return matched_files[0]


def _prepare_apply_chat_template_tools_and_messages(
    messages: list[dict[str, Any]],
    tools: list[dict[str, Any]] | None = None,
    continue_final_message: bool = False,

View on GitHub (pinned to 4a030776a3)

Solutions

Inspect the directory: ls <model_dir> and identify the duplicates matched by the pattern.
Keep only the tokenizer file matching the model's actual format (tekken.json for Tekken models, tokenizer.model.vN for SPM v3/v7) and delete/move the others.
Alternatively point model_id at a clean directory containing a single tokenizer file.

Example fix

# before: model_dir contains tokenizer.model.v3 AND tekken.json
# after: keep only the correct one
rm model_dir/tokenizer.model.v3   # model is Tekken-based
tok = MistralTokenizer.from_pretrained('model_dir')

Defensive patterns

Strategy: validation

Validate before calling

import re
 Mistral_RE = re.compile(r'^tokenizer\.model\.v.*$|^tekken\.json$|^tokenizer\.mm\.model\.v.*$')

def dir_has_single_mistral_tokenizer(path: str) -> bool:
    import os
    matches = [f for f in os.listdir(path) if Mistral_RE.match(f)]
    return len(matches) == 1

Prevention

Package local model dirs with exactly one tokenizer artifact.
Validate downloaded model directories before registering them as local models.
When converting between SPM and Tekken formats, delete the superseded tokenizer file.

When it happens

Trigger: MistralTokenizer.from_pretrained(<local dir>) where the directory contains e.g. both tokenizer.model.v3 and tekken.json — common when someone merged files from two checkpoints or copied a v*-style and a Tekken tokenizer into one folder.

Common situations: Manual model packaging; converting a repo between SPM and Tekken formats without cleaning up; fine-tune artifacts that inherited extra tokenizer files from the base model.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/becba1b2be78bf45. Report an issue: GitHub.