{"record":{"id":"4752642895289bd2","repo":"mudler/LocalAI","slug":"tokenizer-json-not-found-under-model-dir","errorCode":null,"errorMessage":"tokenizer.json not found under {model_dir}","messagePattern":"tokenizer\\.json not found under (.+?)","errorType":"exception","errorClass":"FileNotFoundError","httpStatus":null,"severity":"error","filePath":"backend/python/tinygrad/backend.py","lineNumber":354,"sourceCode":"        # Auto-pick tool parser from options or model family.\n        parser_name = self.options.get(\"tool_parser\") or _auto_tool_parser(self.model_ref, self.llm_config)\n        self.tool_parser = resolve_parser(parser_name)\n\n    def _load_tokenizer_for_dir(self, model_dir: Path, gguf_kv: Optional[dict]) -> None:\n        \"\"\"Load HF tokenizer + chat template + EOS ids from a model directory.\n\n        Falls back to apps.llm's `SimpleTokenizer.from_gguf_kv` when there\n        is no `tokenizer.json` sidecar (single-file GGUF, no HF repo).\n        \"\"\"\n        tokenizer_json = model_dir / \"tokenizer.json\"\n        if tokenizer_json.exists():\n            from tokenizers import Tokenizer as HFTokenizer\n            self.llm_tokenizer = HFTokenizer.from_file(str(tokenizer_json))\n        elif gguf_kv is not None:\n            from tinygrad.apps.llm import SimpleTokenizer\n            self.llm_tokenizer = SimpleTokenizer.from_gguf_kv(gguf_kv)\n        else:\n            raise FileNotFoundError(f\"tokenizer.json not found under {model_dir}\")\n\n        tok_cfg_path = model_dir / \"tokenizer_config.json\"\n        if tok_cfg_path.exists():\n            with open(tok_cfg_path) as fp:\n                tok_cfg = json.load(fp)\n            self.chat_template = tok_cfg.get(\"chat_template\")\n\n        self.llm_eos_ids = []\n        for cfg_name in (\"generation_config.json\", \"config.json\"):\n            cfg_path = model_dir / cfg_name\n            if not cfg_path.exists():\n                continue\n            with open(cfg_path) as fp:\n                cfg = json.load(fp)\n            eos = cfg.get(\"eos_token_id\")\n            if isinstance(eos, list):\n                self.llm_eos_ids.extend(int(x) for x in eos)\n            elif isinstance(eos, int):","sourceCodeStart":336,"sourceCodeEnd":372,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/backend/python/tinygrad/backend.py#L336-L372","documentation":"_load_tokenizer_for_dir prefers a sidecar tokenizer.json (HF tokenizers), falls back to SimpleTokenizer.from_gguf_kv when GGUF metadata is available, and only raises when neither exists — i.e. the directory has neither a tokenizer.json nor GGUF kv to build one from.","triggerScenarios":"Model directory lacks tokenizer.json and the load path did not come from a GGUF file (so gguf_kv is None); snapshot filtered out tokenizer files; pointing at a repo that only ships tokenizer.model (sentencepiece) which this loader does not consume.","commonSituations":"allow_patterns omitting tokenizer.json; older/Gemma-style repos without a converted tokenizer.json; copying a model dir without tokenizer artifacts.","solutions":["Ensure tokenizer.json exists in the model directory; download it from the HF repo if missing.","Add 'tokenizer.json' (and tokenizer_config.json) to any snapshot_download allow_patterns.","If the repo only has a sentencepiece tokenizer.model, use a revision/convert export that includes tokenizer.json."],"exampleFix":"# before: allow_patterns without tokenizer files\nallow_patterns=[\"config.json\", \"*.safetensors\"]\n# after\nallow_patterns=[\"config.json\", \"tokenizer.json\", \"tokenizer_config.json\", \"*.safetensors\"]","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\ndef tokenizer_available(model_dir: str) -> bool:\n    return (Path(model_dir) / \"tokenizer.json\").is_file()","typeGuard":null,"tryCatchPattern":"try:\n    self._load_tokenizer_for_dir(d, gguf_kv=kv)\nexcept FileNotFoundError:\n    from huggingface_hub import hf_hub_download\n    hf_hub_download(repo_id, \"tokenizer.json\", local_dir=d)\n    self._load_tokenizer_for_dir(d, gguf_kv=kv)","preventionTips":["Always allow-pattern tokenizer.json in downloads.","Prefer GGUF files with embedded tokenizer metadata when sidecar files are unreliable.","Check the repo page for tokenizer.json before choosing a model."],"tags":["tinygrad","tokenizer","file-not-found","huggingface","localai"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}