invoke-ai/InvokeAI · error · NotAMatchError

standalone Qwen3-VL encoder directory does not contain token

Error message

standalone Qwen3-VL encoder directory does not contain tokenizer files

What it means

Raised as a NotAMatchError by Qwen3VLEncoder_Qwen3VLEncoder_Config.from_model_on_disk when a candidate directory passes the config.json and model-weights checks but the tokenizer location lacks tokenizer.json or the vocab.json+merges.txt pair. InvokeAI only classifies a directory as a standalone Qwen3-VL encoder if it can find a usable tokenizer alongside the weights, because the encoder loader needs it for text conditioning.

Source

Thrown at invokeai/backend/model_manager/configs/qwen3_vl_encoder.py:151

            },
        )
        _validate_krea2_qwen3_vl_config(expected_config_path)

        if config_path_nested.exists():
            weights_path = mod.path / "text_encoder"
            tokenizer_path = mod.path / "tokenizer"
        else:
            weights_path = mod.path
            tokenizer_path = mod.path

        has_weights = _has_complete_pretrained_weights(weights_path)
        has_tokenizer = (tokenizer_path / "tokenizer.json").exists() or (
            (tokenizer_path / "vocab.json").exists() and (tokenizer_path / "merges.txt").exists()
        )
        if not has_weights:
            raise NotAMatchError("standalone Qwen3-VL encoder directory does not contain model weights")
        if not has_tokenizer:
            raise NotAMatchError("standalone Qwen3-VL encoder directory does not contain tokenizer files")

        return cls(**override_fields)


def _is_qwen3_vl_encoder_state_dict(state_dict: dict[str | int, Any]) -> bool:
    """True for a single-file Qwen3-VL encoder: a Qwen3 text decoder PLUS a visual tower.

    The visual tower (``visual.*`` / ``model.visual.*``) distinguishes Qwen3-VL from the text-only
    ``Qwen3Encoder`` (Z-Image / FLUX.2 Klein), which has ``model.layers.*`` but no visual tower.
    """
    str_keys = [k for k in state_dict if isinstance(k, str)]
    has_text_decoder = any(".layers." in k and ("model." in k or k.startswith("layers.")) for k in str_keys)
    has_visual_tower = any(k.startswith(("visual.", "model.visual.")) or ".visual." in k for k in str_keys)
    return has_text_decoder and has_visual_tower


class Qwen3VLEncoder_Checkpoint_Config(Checkpoint_Config_Base, Config_Base):
    """Configuration for a single-file Qwen3-VL text encoder checkpoint (e.g. ComfyUI ``qwen3vl_4b_*``).

View on GitHub (pinned to 0b6a024f2f)

Solutions

Check which layout was detected: if text_encoder/config.json exists, place tokenizer files in <root>/tokenizer/; otherwise place them at the directory root.
Copy tokenizer.json (or vocab.json plus merges.txt) from the matching HuggingFace repo (e.g. Qwen/Qwen3-VL-4B-Instruct) into the expected tokenizer location.
Re-download the model with git lfs or huggingface-cli download so no tokenizer assets are skipped, then rescan.
If you intended a full pipeline instead, point InvokeAI at the parent directory containing model_index.json so it is matched as a Main model.

Example fix

// before (directory layout)
my-encoder/
  config.json
  model.safetensors

// after
my-encoder/
  config.json
  model.safetensors
  tokenizer.json        # copied from Qwen/Qwen3-VL-4B-Instruct
  tokenizer_config.json

Defensive patterns

Strategy: validation

Validate before calling

from pathlib import Path

def has_standalone_qwen3vl_tokenizer(root: Path) -> bool:
    nested = root / "text_encoder" / "config.json"
    tokenizer_dir = root / "tokenizer" if nested.exists() else root
    has_json = (tokenizer_dir / "tokenizer.json").exists()
    has_vocab_merges = (tokenizer_dir / "vocab.json").exists() and (tokenizer_dir / "merges.txt").exists()
    return has_json or has_vocab_merges

assert has_standalone_qwen3vl_tokenizer(Path("/path/to/model")), "tokenizer files missing"

Type guard

from pathlib import Path

def is_tokenizer_complete(d: Path) -> bool:
    return (
        (d / "tokenizer.json").is_file()
        or ((d / "vocab.json").is_file() and (d / "merges.txt").is_file())
    )

Try / catch

try:
    config = invokeai_model_manager.probe(path)
except NotAMatchError as e:
    if "does not contain tokenizer files" in str(e):
        download_tokenizer_from_hub("Qwen/Qwen3-VL-4B-Instruct", dest=path / "tokenizer")
    else:
        raise

Prevention

Always download encoder repos with huggingface-cli download so tokenizer assets are included.
Before importing, run a quick script checking for tokenizer.json (or vocab.json + merges.txt) in the expected location.
Remember the layout rule: nested text_encoder/ config means tokenizer goes in tokenizer/; root config.json means tokenizer files sit at the root.
Don't strip tokenizer files when copying model folders manually.

When it happens

Trigger: Calling model identification (model probe / import) on a directory where text_encoder/ holds weights and config.json but the tokenizer/ subfolder is absent or empty, or a standalone root layout where config.json and weights exist at the root but no tokenizer.json / vocab.json+merges.txt is present next to them.

Common situations: Partial or interrupted HuggingFace download (tokenizer files skipped or in .cache only); manually copying only the weights folder out of a repo; downloading a repo that keeps tokenizer files in a differently named folder (e.g. tokenizer/ missing while files sit elsewhere); stripped-down model releases that omit tokenizer assets.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/d8c95cc918e40b0d. Report an issue: GitHub.