invoke-ai/InvokeAI · error · ValueError

Could not find attention/mlp weights in state dict to determ

Error message

Could not find attention/mlp weights in state dict to determine configuration

What it means

The GGUF loader detects the attention configuration (head counts, dimensions) from the shapes of model.layers.0.self_attn.q_proj.weight, k_proj.weight and model.layers.0.mlp.gate_proj.weight. If any of these keys are missing it cannot configure the model and raises this ValueError.

Source

Thrown at invokeai/backend/model_manager/load/model_loaders/z_image.py:1266

        # Detect attention configuration from layer weights
        # IMPORTANT: Use layer 1 (not layer 0) because some models like FLUX 2 Klein have a special
        # first layer with different dimensions (input projection layer) while the rest of the
        # transformer layers have a different hidden_size. Using a middle layer ensures we get
        # the representative hidden_size for the bulk of the model.
        # Fall back to layer 0 if layer 1 doesn't exist.
        q_proj_weight = sd.get("model.layers.1.self_attn.q_proj.weight")
        k_proj_weight = sd.get("model.layers.1.self_attn.k_proj.weight")
        gate_proj_weight = sd.get("model.layers.1.mlp.gate_proj.weight")

        # Fall back to layer 0 if layer 1 doesn't exist (single-layer model edge case)
        if q_proj_weight is None:
            q_proj_weight = sd.get("model.layers.0.self_attn.q_proj.weight")
            k_proj_weight = sd.get("model.layers.0.self_attn.k_proj.weight")
            gate_proj_weight = sd.get("model.layers.0.mlp.gate_proj.weight")

        if q_proj_weight is None or k_proj_weight is None or gate_proj_weight is None:
            raise ValueError("Could not find attention/mlp weights in state dict to determine configuration")

        # Handle GGMLTensor shape access
        q_shape = q_proj_weight.shape if hasattr(q_proj_weight, "shape") else q_proj_weight.tensor_shape
        k_shape = k_proj_weight.shape if hasattr(k_proj_weight, "shape") else k_proj_weight.tensor_shape
        gate_shape = gate_proj_weight.shape if hasattr(gate_proj_weight, "shape") else gate_proj_weight.tensor_shape

        # Calculate dimensions from actual weights
        # IMPORTANT: Use hidden_size from k_proj input dimension (not q_proj or embed_tokens).
        # Some models (like FLUX 2 Klein) have unusual architectures where:
        # - embed_tokens has a larger dimension (e.g., 2560)
        # - q_proj may have a larger input dimension for query expansion
        # - k_proj/v_proj have the actual transformer hidden_size (e.g., 1280)
        # Using k_proj ensures we get the correct internal hidden_size.
        head_dim = 128  # Standard head dimension for Qwen3 models
        hidden_size = k_shape[1]  # Use k_proj input dim as the hidden_size
        num_attention_heads = q_shape[0] // head_dim
        num_kv_heads = k_shape[0] // head_dim
        intermediate_size = gate_shape[0]

View on GitHub (pinned to 0b6a024f2f)

Solutions

Verify the GGUF is actually the Qwen3-based Z-Image text encoder (check keys with gguf-dump).
Re-export the GGUF with HF-style key names (model.layers.N.self_attn.q_proj.weight etc.).
Extend the loader's key lookup to also accept alternative names (e.g. 'blk.0.attn_q.weight' -> q_proj) if your conversion pipeline uses them.
Use the safetensors checkpoint directly if GGUF conversion keeps renaming keys.

Example fix

// before: llama.cpp-style keys in GGUF
'blk.0.attn_q.weight', 'blk.0.attn_k.weight', 'blk.0.ffn_gate.weight'
// after: re-export with HF mapping
'model.layers.0.self_attn.q_proj.weight', 'model.layers.0.self_attn.k_proj.weight', 'model.layers.0.mlp.gate_proj.weight'

Defensive patterns

Strategy: validation

Validate before calling

required = {"model.layers.0.self_attn.q_proj.weight",
           "model.layers.0.self_attn.k_proj.weight",
           "model.layers.0.mlp.gate_proj.weight"}
names = {t.name for t in gguf.GGUFReader(path).tensors}
missing = required - names
if missing:
    raise ValueError(f"{path} missing Qwen3 projection keys: {missing}")

Type guard

def is_qwen3_encoder_gguf(path: str) -> bool:
    try:
        names = {t.name for t in gguf.GGUFReader(path).tensors}
        return any(n.startswith("model.layers.0.self_attn.q_proj") for n in names)
    except Exception:
        return False

Try / catch

try:
    model = load_text_encoder(cfg)
except ValueError as e:
    if "attention/mlp weights" in str(e):
        fix_key_mapping_or_use_safetensors(cfg.path)
    else:
        raise

Prevention

Convert GGUFs with HF key-name mapping enabled.
gguf-dump layer-0 keys before installing a converted model.
Only use models explicitly released for Z-Image text encoding.
Keep safetensors fallback available for unconvertible architectures.

When it happens

Trigger: Loading a GGUF whose layer-0 weights use different names (different transformer architecture, e.g. fused QKV projections, llama.cpp-style naming like blk.0.attn_q.weight, or an encoder with no MLP gate_proj).

Common situations: Using a GGUF of a non-Qwen3 model as the Z-Image text encoder; GGUF converted with llama.cpp naming conventions; pruned/edited checkpoints with removed or renamed projections.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/311158ab57afc154. Report an issue: GitHub.