{"record":{"id":"311158ab57afc154","repo":"invoke-ai/InvokeAI","slug":"could-not-find-attention-mlp-weights-in-state-dict-311158","errorCode":null,"errorMessage":"Could not find attention/mlp weights in state dict to determine configuration","messagePattern":"Could not find attention/mlp weights in state dict to determine configuration","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/model_manager/load/model_loaders/z_image.py","lineNumber":1266,"sourceCode":"\n        # Detect attention configuration from layer weights\n        # IMPORTANT: Use layer 1 (not layer 0) because some models like FLUX 2 Klein have a special\n        # first layer with different dimensions (input projection layer) while the rest of the\n        # transformer layers have a different hidden_size. Using a middle layer ensures we get\n        # the representative hidden_size for the bulk of the model.\n        # Fall back to layer 0 if layer 1 doesn't exist.\n        q_proj_weight = sd.get(\"model.layers.1.self_attn.q_proj.weight\")\n        k_proj_weight = sd.get(\"model.layers.1.self_attn.k_proj.weight\")\n        gate_proj_weight = sd.get(\"model.layers.1.mlp.gate_proj.weight\")\n\n        # Fall back to layer 0 if layer 1 doesn't exist (single-layer model edge case)\n        if q_proj_weight is None:\n            q_proj_weight = sd.get(\"model.layers.0.self_attn.q_proj.weight\")\n            k_proj_weight = sd.get(\"model.layers.0.self_attn.k_proj.weight\")\n            gate_proj_weight = sd.get(\"model.layers.0.mlp.gate_proj.weight\")\n\n        if q_proj_weight is None or k_proj_weight is None or gate_proj_weight is None:\n            raise ValueError(\"Could not find attention/mlp weights in state dict to determine configuration\")\n\n        # Handle GGMLTensor shape access\n        q_shape = q_proj_weight.shape if hasattr(q_proj_weight, \"shape\") else q_proj_weight.tensor_shape\n        k_shape = k_proj_weight.shape if hasattr(k_proj_weight, \"shape\") else k_proj_weight.tensor_shape\n        gate_shape = gate_proj_weight.shape if hasattr(gate_proj_weight, \"shape\") else gate_proj_weight.tensor_shape\n\n        # Calculate dimensions from actual weights\n        # IMPORTANT: Use hidden_size from k_proj input dimension (not q_proj or embed_tokens).\n        # Some models (like FLUX 2 Klein) have unusual architectures where:\n        # - embed_tokens has a larger dimension (e.g., 2560)\n        # - q_proj may have a larger input dimension for query expansion\n        # - k_proj/v_proj have the actual transformer hidden_size (e.g., 1280)\n        # Using k_proj ensures we get the correct internal hidden_size.\n        head_dim = 128  # Standard head dimension for Qwen3 models\n        hidden_size = k_shape[1]  # Use k_proj input dim as the hidden_size\n        num_attention_heads = q_shape[0] // head_dim\n        num_kv_heads = k_shape[0] // head_dim\n        intermediate_size = gate_shape[0]","sourceCodeStart":1248,"sourceCodeEnd":1284,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/model_manager/load/model_loaders/z_image.py#L1248-L1284","documentation":"The GGUF loader detects the attention configuration (head counts, dimensions) from the shapes of model.layers.0.self_attn.q_proj.weight, k_proj.weight and model.layers.0.mlp.gate_proj.weight. If any of these keys are missing it cannot configure the model and raises this ValueError.","triggerScenarios":"Loading a GGUF whose layer-0 weights use different names (different transformer architecture, e.g. fused QKV projections, llama.cpp-style naming like blk.0.attn_q.weight, or an encoder with no MLP gate_proj).","commonSituations":"Using a GGUF of a non-Qwen3 model as the Z-Image text encoder; GGUF converted with llama.cpp naming conventions; pruned/edited checkpoints with removed or renamed projections.","solutions":["Verify the GGUF is actually the Qwen3-based Z-Image text encoder (check keys with gguf-dump).","Re-export the GGUF with HF-style key names (model.layers.N.self_attn.q_proj.weight etc.).","Extend the loader's key lookup to also accept alternative names (e.g. 'blk.0.attn_q.weight' -> q_proj) if your conversion pipeline uses them.","Use the safetensors checkpoint directly if GGUF conversion keeps renaming keys."],"exampleFix":"// before: llama.cpp-style keys in GGUF\n'blk.0.attn_q.weight', 'blk.0.attn_k.weight', 'blk.0.ffn_gate.weight'\n// after: re-export with HF mapping\n'model.layers.0.self_attn.q_proj.weight', 'model.layers.0.self_attn.k_proj.weight', 'model.layers.0.mlp.gate_proj.weight'","handlingStrategy":"validation","validationCode":"required = {\"model.layers.0.self_attn.q_proj.weight\",\n           \"model.layers.0.self_attn.k_proj.weight\",\n           \"model.layers.0.mlp.gate_proj.weight\"}\nnames = {t.name for t in gguf.GGUFReader(path).tensors}\nmissing = required - names\nif missing:\n    raise ValueError(f\"{path} missing Qwen3 projection keys: {missing}\")","typeGuard":"def is_qwen3_encoder_gguf(path: str) -> bool:\n    try:\n        names = {t.name for t in gguf.GGUFReader(path).tensors}\n        return any(n.startswith(\"model.layers.0.self_attn.q_proj\") for n in names)\n    except Exception:\n        return False","tryCatchPattern":"try:\n    model = load_text_encoder(cfg)\nexcept ValueError as e:\n    if \"attention/mlp weights\" in str(e):\n        fix_key_mapping_or_use_safetensors(cfg.path)\n    else:\n        raise","preventionTips":["Convert GGUFs with HF key-name mapping enabled.","gguf-dump layer-0 keys before installing a converted model.","Only use models explicitly released for Z-Image text encoding.","Keep safetensors fallback available for unconvertible architectures."],"tags":["gguf","model-loading","missing-tensor","key-mapping"],"backgroundTag":"missing-tensor-in-state-dict","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}