{"record":{"id":"d8c95cc918e40b0d","repo":"invoke-ai/InvokeAI","slug":"standalone-qwen3-vl-encoder-directory-does-not-con-d8c95c","errorCode":null,"errorMessage":"standalone Qwen3-VL encoder directory does not contain tokenizer files","messagePattern":"standalone Qwen3-VL encoder directory does not contain tokenizer files","errorType":"validation","errorClass":"NotAMatchError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/model_manager/configs/qwen3_vl_encoder.py","lineNumber":151,"sourceCode":"            },\n        )\n        _validate_krea2_qwen3_vl_config(expected_config_path)\n\n        if config_path_nested.exists():\n            weights_path = mod.path / \"text_encoder\"\n            tokenizer_path = mod.path / \"tokenizer\"\n        else:\n            weights_path = mod.path\n            tokenizer_path = mod.path\n\n        has_weights = _has_complete_pretrained_weights(weights_path)\n        has_tokenizer = (tokenizer_path / \"tokenizer.json\").exists() or (\n            (tokenizer_path / \"vocab.json\").exists() and (tokenizer_path / \"merges.txt\").exists()\n        )\n        if not has_weights:\n            raise NotAMatchError(\"standalone Qwen3-VL encoder directory does not contain model weights\")\n        if not has_tokenizer:\n            raise NotAMatchError(\"standalone Qwen3-VL encoder directory does not contain tokenizer files\")\n\n        return cls(**override_fields)\n\n\ndef _is_qwen3_vl_encoder_state_dict(state_dict: dict[str | int, Any]) -> bool:\n    \"\"\"True for a single-file Qwen3-VL encoder: a Qwen3 text decoder PLUS a visual tower.\n\n    The visual tower (``visual.*`` / ``model.visual.*``) distinguishes Qwen3-VL from the text-only\n    ``Qwen3Encoder`` (Z-Image / FLUX.2 Klein), which has ``model.layers.*`` but no visual tower.\n    \"\"\"\n    str_keys = [k for k in state_dict if isinstance(k, str)]\n    has_text_decoder = any(\".layers.\" in k and (\"model.\" in k or k.startswith(\"layers.\")) for k in str_keys)\n    has_visual_tower = any(k.startswith((\"visual.\", \"model.visual.\")) or \".visual.\" in k for k in str_keys)\n    return has_text_decoder and has_visual_tower\n\n\nclass Qwen3VLEncoder_Checkpoint_Config(Checkpoint_Config_Base, Config_Base):\n    \"\"\"Configuration for a single-file Qwen3-VL text encoder checkpoint (e.g. ComfyUI ``qwen3vl_4b_*``).","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/model_manager/configs/qwen3_vl_encoder.py#L133-L169","documentation":"Raised as a NotAMatchError by Qwen3VLEncoder_Qwen3VLEncoder_Config.from_model_on_disk when a candidate directory passes the config.json and model-weights checks but the tokenizer location lacks tokenizer.json or the vocab.json+merges.txt pair. InvokeAI only classifies a directory as a standalone Qwen3-VL encoder if it can find a usable tokenizer alongside the weights, because the encoder loader needs it for text conditioning.","triggerScenarios":"Calling model identification (model probe / import) on a directory where text_encoder/ holds weights and config.json but the tokenizer/ subfolder is absent or empty, or a standalone root layout where config.json and weights exist at the root but no tokenizer.json / vocab.json+merges.txt is present next to them.","commonSituations":"Partial or interrupted HuggingFace download (tokenizer files skipped or in .cache only); manually copying only the weights folder out of a repo; downloading a repo that keeps tokenizer files in a differently named folder (e.g. tokenizer/ missing while files sit elsewhere); stripped-down model releases that omit tokenizer assets.","solutions":["Check which layout was detected: if text_encoder/config.json exists, place tokenizer files in <root>/tokenizer/; otherwise place them at the directory root.","Copy tokenizer.json (or vocab.json plus merges.txt) from the matching HuggingFace repo (e.g. Qwen/Qwen3-VL-4B-Instruct) into the expected tokenizer location.","Re-download the model with git lfs or huggingface-cli download so no tokenizer assets are skipped, then rescan.","If you intended a full pipeline instead, point InvokeAI at the parent directory containing model_index.json so it is matched as a Main model."],"exampleFix":"// before (directory layout)\nmy-encoder/\n  config.json\n  model.safetensors\n\n// after\nmy-encoder/\n  config.json\n  model.safetensors\n  tokenizer.json        # copied from Qwen/Qwen3-VL-4B-Instruct\n  tokenizer_config.json","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\ndef has_standalone_qwen3vl_tokenizer(root: Path) -> bool:\n    nested = root / \"text_encoder\" / \"config.json\"\n    tokenizer_dir = root / \"tokenizer\" if nested.exists() else root\n    has_json = (tokenizer_dir / \"tokenizer.json\").exists()\n    has_vocab_merges = (tokenizer_dir / \"vocab.json\").exists() and (tokenizer_dir / \"merges.txt\").exists()\n    return has_json or has_vocab_merges\n\nassert has_standalone_qwen3vl_tokenizer(Path(\"/path/to/model\")), \"tokenizer files missing\"","typeGuard":"from pathlib import Path\n\ndef is_tokenizer_complete(d: Path) -> bool:\n    return (\n        (d / \"tokenizer.json\").is_file()\n        or ((d / \"vocab.json\").is_file() and (d / \"merges.txt\").is_file())\n    )","tryCatchPattern":"try:\n    config = invokeai_model_manager.probe(path)\nexcept NotAMatchError as e:\n    if \"does not contain tokenizer files\" in str(e):\n        download_tokenizer_from_hub(\"Qwen/Qwen3-VL-4B-Instruct\", dest=path / \"tokenizer\")\n    else:\n        raise","preventionTips":["Always download encoder repos with huggingface-cli download so tokenizer assets are included.","Before importing, run a quick script checking for tokenizer.json (or vocab.json + merges.txt) in the expected location.","Remember the layout rule: nested text_encoder/ config means tokenizer goes in tokenizer/; root config.json means tokenizer files sit at the root.","Don't strip tokenizer files when copying model folders manually."],"tags":["invokeai","model-import","tokenizer","model-manager","not-a-match"],"backgroundTag":"missing-model-files","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}