mudler/LocalAI · error · FileNotFoundError

tokenizer.json not found under {model_dir}

Error message

tokenizer.json not found under {model_dir}

What it means

_load_tokenizer_for_dir prefers a sidecar tokenizer.json (HF tokenizers), falls back to SimpleTokenizer.from_gguf_kv when GGUF metadata is available, and only raises when neither exists — i.e. the directory has neither a tokenizer.json nor GGUF kv to build one from.

Source

Thrown at backend/python/tinygrad/backend.py:354

        # Auto-pick tool parser from options or model family.
        parser_name = self.options.get("tool_parser") or _auto_tool_parser(self.model_ref, self.llm_config)
        self.tool_parser = resolve_parser(parser_name)

    def _load_tokenizer_for_dir(self, model_dir: Path, gguf_kv: Optional[dict]) -> None:
        """Load HF tokenizer + chat template + EOS ids from a model directory.

        Falls back to apps.llm's `SimpleTokenizer.from_gguf_kv` when there
        is no `tokenizer.json` sidecar (single-file GGUF, no HF repo).
        """
        tokenizer_json = model_dir / "tokenizer.json"
        if tokenizer_json.exists():
            from tokenizers import Tokenizer as HFTokenizer
            self.llm_tokenizer = HFTokenizer.from_file(str(tokenizer_json))
        elif gguf_kv is not None:
            from tinygrad.apps.llm import SimpleTokenizer
            self.llm_tokenizer = SimpleTokenizer.from_gguf_kv(gguf_kv)
        else:
            raise FileNotFoundError(f"tokenizer.json not found under {model_dir}")

        tok_cfg_path = model_dir / "tokenizer_config.json"
        if tok_cfg_path.exists():
            with open(tok_cfg_path) as fp:
                tok_cfg = json.load(fp)
            self.chat_template = tok_cfg.get("chat_template")

        self.llm_eos_ids = []
        for cfg_name in ("generation_config.json", "config.json"):
            cfg_path = model_dir / cfg_name
            if not cfg_path.exists():
                continue
            with open(cfg_path) as fp:
                cfg = json.load(fp)
            eos = cfg.get("eos_token_id")
            if isinstance(eos, list):
                self.llm_eos_ids.extend(int(x) for x in eos)
            elif isinstance(eos, int):

View on GitHub (pinned to 44413a9d06)

Solutions

1. Ensure tokenizer.json exists in the model directory; download it from the HF repo if missing.
2. Add 'tokenizer.json' (and tokenizer_config.json) to any snapshot_download allow_patterns.
3. If the repo only has a sentencepiece tokenizer.model, use a revision/convert export that includes tokenizer.json.

Example fix

# before: allow_patterns without tokenizer files
allow_patterns=["config.json", "*.safetensors"]
# after
allow_patterns=["config.json", "tokenizer.json", "tokenizer_config.json", "*.safetensors"]

from pathlib import Path

def tokenizer_available(model_dir: str) -> bool:
    return (Path(model_dir) / "tokenizer.json").is_file()

try:
    self._load_tokenizer_for_dir(d, gguf_kv=kv)
except FileNotFoundError:
    from huggingface_hub import hf_hub_download
    hf_hub_download(repo_id, "tokenizer.json", local_dir=d)
    self._load_tokenizer_for_dir(d, gguf_kv=kv)

When it happens

Trigger: Model directory lacks tokenizer.json and the load path did not come from a GGUF file (so gguf_kv is None); snapshot filtered out tokenizer files; pointing at a repo that only ships tokenizer.model (sentencepiece) which this loader does not consume.

Common situations: allow_patterns omitting tokenizer.json; older/Gemma-style repos without a converted tokenizer.json; copying a model dir without tokenizer artifacts.

Related errors

• Model not found: {model_ref}
• Expected HF model directory, got file: {model_path}
• config.json not found under {model_path}
• No safetensors weights found under {model_dir}
• ONNX model not found: {onnx_path}

AI-assisted analysis of mudler/LocalAI@44413a9d06 (2026-08-15). Data as JSON: /api/errors/4752642895289bd2. Report an issue: GitHub.