invoke-ai/InvokeAI · error · ValueError

Could not find attention/mlp weights to determine configurat

Error message

Could not find attention/mlp weights to determine configuration

What it means

After reading embed_tokens, the loader inspects layers.0's q_proj/k_proj (attention) and gate_proj (MLP) weights to infer head counts and intermediate size for the Qwen3 config. If none of these tensors exist under the expected names, the architecture cannot be detected and this ValueError is raised.

Source

Thrown at invokeai/backend/model_manager/load/model_loaders/z_image.py:1069

            logger.info("Detected Qwen3-8B variant")
            hidden_size = 4096
            num_attention_heads = 32
            num_kv_heads = 8
            intermediate_size = 12288
            head_dim = 128
            max_position_embeddings = 40960
        else:
            # Unknown variant - try to detect from weights
            logger.warning(
                f"Unknown Qwen3 variant: embed_hidden_size={embed_hidden_size}, layers={layer_count}. "
                "Attempting to detect configuration from weights..."
            )
            q_proj_weight = sd.get("model.layers.0.self_attn.q_proj.weight")
            k_proj_weight = sd.get("model.layers.0.self_attn.k_proj.weight")
            gate_proj_weight = sd.get("model.layers.0.mlp.gate_proj.weight")

            if q_proj_weight is None or k_proj_weight is None or gate_proj_weight is None:
                raise ValueError("Could not find attention/mlp weights to determine configuration")

            hidden_size = embed_hidden_size
            head_dim = 128
            num_attention_heads = q_proj_weight.shape[0] // head_dim
            num_kv_heads = k_proj_weight.shape[0] // head_dim
            intermediate_size = gate_proj_weight.shape[0]
            max_position_embeddings = 40960

        logger.info(
            f"Qwen3 config: hidden_size={hidden_size}, layers={layer_count}, "
            f"heads={num_attention_heads}, kv_heads={num_kv_heads}, intermediate={intermediate_size}"
        )

        # Create Qwen3 config
        qwen_config = Qwen3Config(
            vocab_size=vocab_size,
            hidden_size=hidden_size,
            intermediate_size=intermediate_size,

View on GitHub (pinned to 0b6a024f2f)

Solutions

Load a checkpoint with standard HF Qwen3 key naming (separate q_proj/k_proj/gate_proj per layer).
If keys are only prefixed, strip the prefix (e.g. text_model.) before/while loading so the expected names appear at the root.
If the checkpoint uses fused attention (qkv_proj), convert it to split q/k/v projections with a conversion script.
Dump sd.keys() and confirm model.layers.0.self_attn.q_proj.weight exists before loading.

Example fix

// before
sd = {"model.layers.0.self_attn.qkv_proj.weight": ...}  # fused keys -> error
// after
sd = {"model.layers.0.self_attn.q_proj.weight": ..., "model.layers.0.self_attn.k_proj.weight": ..., "model.layers.0.mlp.gate_proj.weight": ...}

Defensive patterns

Strategy: validation

Validate before calling

REQUIRED = {"model.layers.0.self_attn.q_proj.weight",
            "model.layers.0.self_attn.k_proj.weight",
            "model.layers.0.mlp.gate_proj.weight"}
with safe_open(path, framework="pt") as f:
    missing = REQUIRED - set(f.keys())
if missing:
    raise ValueError(f"Missing expected Qwen3 layer keys: {missing}")

Try / catch

try:
    model = loader._load_model(config, submodel_type=SubModelType.TextEncoder)
except ValueError as e:
    if "attention/mlp weights" in str(e):
        raise CheckpointFormatError("Fused/renamed layer keys; convert to HF Qwen3 naming") from e
    raise

Prevention

Avoid checkpoints with fused qkv_proj keys; convert to split q/k/v projections first.
Strip exporter prefixes (e.g. text_model.) so keys match HF Qwen3 names.
Dump state-dict keys and diff against a known-good Qwen3 checkpoint before loading.
Pin to officially supported Z-Image text-encoder exports.

When it happens

Trigger: The state dict has model.embed_tokens.weight but lacks model.layers.0.self_attn.q_proj.weight, k_proj.weight, or mlp.gate_proj.weight — e.g. layers stored with fused/packed attention names, a different layer indexing, or an export that dropped early-layer keys.

Common situations: Custom or quantized exports with fused QKV projections (qkv_proj) instead of separate q_proj/k_proj; checkpoints with a top-level prefix such as text_model.layers...; converted checkpoints where layer 0 was renumbered or pruned.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/88cbfce27ecfec03. Report an issue: GitHub.