zylon-ai/private-gpt · error · OSError

Found {len(matched_files)} files matching the pattern: {file

Error message

Found {len(matched_files)} files matching the pattern: {file_pattern}. Make sure that a Mistral tokenizer is present in {files}.

What it means

The companion case of the ambiguous-file error: find_tokenizer_file matched zero files against ^tokenizer\.model\.v.*$|^tekken\.json$|^tokenizer\.mm\.model\.v.*$ in the given directory list, so there is no Mistral tokenizer to load and it raises OSError.

Source

Thrown at private_gpt/components/llm/tokenizers/mistral.py:196

    return []


def find_tokenizer_file(files: list[str]) -> str:
    """Find the Mistral tokenizer file from a list of repository files."""
    file_pattern = re.compile(
        r"^tokenizer\.model\.v.*$|^tekken\.json$|^tokenizer\.mm\.model\.v.*$"
    )

    matched_files = [file for file in files if file_pattern.match(file)]
    if len(matched_files) > 1:
        raise OSError(
            f"Found {len(matched_files)} files matching the "
            f"pattern: {file_pattern}. Make sure only one Mistral "
            f"tokenizer is present in {files}."
        )
    elif len(matched_files) == 0:
        raise OSError(
            f"Found {len(matched_files)} files matching the "
            f"pattern: {file_pattern}. Make sure that a Mistral "
            f"tokenizer is present in {files}."
        )

    return matched_files[0]


def _prepare_apply_chat_template_tools_and_messages(
    messages: list[dict[str, Any]],
    tools: list[dict[str, Any]] | None = None,
    continue_final_message: bool = False,
    add_generation_prompt: bool = False,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]] | None]:
    """Prepare messages and tools for Mistral's chat template format.

    Handles validation and formatting of messages and tools to ensure
    compatibility with Mistral's requirements.

View on GitHub (pinned to 4a030776a3)

Solutions

1. Check the directory/repo file list in the error and confirm which tokenizer file format the model actually ships.
2. Use a model repo that contains a Mistral tokenizer file (tekken.json or tokenizer.model.vN), e.g. official mistralai repos.
3. If the model only has HF tokenizer.json, use tokenizer_mode='huggingface' instead of 'mistral'.

Example fix

# before: model_dir only has tokenizer.json (HF format)
tok = MistralTokenizer.from_pretrained('model_dir')

# after
tok = TokenizerRegistry.get_tokenizer('huggingface', model_id='model_dir')

def dir_has_mistral_tokenizer(path: str) -> bool:
    import os, re
    pat = re.compile(r'^tokenizer\.model\.v.*$|^tekken\.json$|^tokenizer\.mm\.model\.v.*$')
    return any(pat.match(f) for f in os.listdir(path))

When it happens

Trigger: MistralTokenizer.from_pretrained(<local dir>) on a directory containing only HF-style files (tokenizer.json / tokenizer_config.json) and no Mistral-format file; also triggered on HF Hub repos (list_repo_files) that lack the expected file.

Common situations: Pointing the mistral tokenizer mode at a generic HF-converted model; incomplete downloads; directory listing that includes files under a subfolder (pattern is anchored to the bare filename).

Related errors

• Found {len(matched_files)} files matching the pattern: {file
• Mistral tokenizer dependencies are not installed. Install wi
• Mistral tokenizer must be in test mode. Set `mode=Validation
• Unsupported tokenizer: {type(self.tokenizer)}
• Empty response from Mistral tokenizer

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/b9489cbfd23aa20e. Report an issue: GitHub.