{"record":{"id":"7c3d5b8a0b611a5d","repo":"invoke-ai/InvokeAI","slug":"directory-does-not-contain-gemma2-tokenizer-files","errorCode":null,"errorMessage":"directory does not contain Gemma2 tokenizer files (tokenizer.json/tokenizer.model)","messagePattern":"directory does not contain Gemma2 tokenizer files \\(tokenizer\\.json/tokenizer\\.model\\)","errorType":"validation","errorClass":"NotAMatchError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/model_manager/configs/gemma2_encoder.py","lineNumber":85,"sourceCode":"            raise NotAMatchError(\"directory looks like a full diffusers pipeline, not a standalone Gemma2 encoder\")\n\n        # Architecture marker is the canonical signal.\n        raise_for_class_name(config_path, {\"Gemma2ForCausalLM\"})\n\n        # Only Gemma-2-2b (2304-dim hidden state) is compatible with PiD's fixed caption projection.\n        # Reject 9B/27B variants here so they are not offered as compatible encoders and then fail with\n        # a matrix-shape error deep inside PiD inference.\n        hidden_size = get_config_dict_or_raise(config_path).get(\"hidden_size\")\n        if hidden_size != _PID_GEMMA_HIDDEN_SIZE:\n            raise NotAMatchError(\n                f\"Gemma2 hidden_size {hidden_size} is incompatible with PiD, which requires \"\n                f\"{_PID_GEMMA_HIDDEN_SIZE} (Gemma-2-2b); 9B/27B variants are not supported.\"\n            )\n\n        # Sanity check that tokenizer files live alongside the model (PiD calls\n        # AutoTokenizer.from_pretrained on the same directory).\n        if not any((mod.path / f).exists() for f in (\"tokenizer.json\", \"tokenizer.model\")):\n            raise NotAMatchError(\"directory does not contain Gemma2 tokenizer files (tokenizer.json/tokenizer.model)\")\n\n        return cls(**override_fields)\n\n\ndef _read_gguf_arch_and_hidden_size(path: Path) -> tuple[str, int | None]:\n    \"\"\"Read (general.architecture, <arch>.embedding_length) from a GGUF file's metadata.\n\n    Raises NotAMatchError if the file is not a readable GGUF or is missing the architecture marker.\n    \"\"\"\n    import gguf\n\n    try:\n        reader = gguf.GGUFReader(path)\n    except Exception as e:\n        raise NotAMatchError(f\"not a readable GGUF file: {e}\") from e\n\n    arch_field = reader.fields.get(\"general.architecture\")\n    if arch_field is None:","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/model_manager/configs/gemma2_encoder.py#L67-L103","documentation":"NotAMatchError raised by Gemma2Encoder_Gemma2Encoder_Config.from_model_on_disk when the model directory contains a valid Gemma2ForCausalLM config.json but no tokenizer.json or tokenizer.model file. PiD later calls AutoTokenizer.from_pretrained on this same directory, so the tokenizer files must live alongside the weights; the config refuses to match a directory that would fail at load time.","triggerScenarios":"Importing a directory that has config.json + weights but the tokenizer files were deleted, not downloaded, or were saved into a sibling subdirectory instead of the model root.","commonSituations":"Manually copying only the safetensors and config.json from a HuggingFace repo; a download tool that skips tokenizer assets; hand-assembled model folders; using save_pretrained on a subdirectory while pointing InvokeAI at the parent.","solutions":["Copy tokenizer.json (or tokenizer.model) and tokenizer_config.json from the original HuggingFace repo into the model directory root","Re-download the full gemma-2-2b-it repo rather than cherry-picking files","Verify with ls: the directory must contain config.json, model*.safetensors, and tokenizer.json/tokenizer.model at the same level"],"exampleFix":"// before\nmodels/gemma-2-2b-it/{config.json, model.safetensors}\n// after\nmodels/gemma-2-2b-it/{config.json, model.safetensors, tokenizer.json, tokenizer_config.json}","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\ndef tokenizer_files_present(model_dir: str | Path) -> bool:\n    d = Path(model_dir)\n    return any((d / f).exists() for f in (\"tokenizer.json\", \"tokenizer.model\"))","typeGuard":"def is_complete_gemma2_dir(p: Path) -> bool:\n    return p.is_dir() and (p / \"config.json\").is_file() and tokenizer_files_present(p)","tryCatchPattern":"try:\n    add_model_and_import(model_dir)\nexcept NotAMatchError as e:\n    if \"tokenizer\" in str(e):\n        print(\"Copy tokenizer.json/tokenizer.model from the HuggingFace repo into the model dir\")","preventionTips":["Download the whole HuggingFace repo (huggingface-cli download) instead of cherry-picking files","After copying models, verify config.json + weights + tokenizer.json/tokenizer.model sit at the same directory level","Keep tokenizer_config.json too — PiD calls AutoTokenizer.from_pretrained on the directory"],"tags":["model-import","tokenizer","missing-file"],"backgroundTag":"missing-tokenizer-files","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}