{"record":{"id":"88cbfce27ecfec03","repo":"invoke-ai/InvokeAI","slug":"could-not-find-attention-mlp-weights-to-determine","errorCode":null,"errorMessage":"Could not find attention/mlp weights to determine configuration","messagePattern":"Could not find attention/mlp weights to determine configuration","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/model_manager/load/model_loaders/z_image.py","lineNumber":1069,"sourceCode":"            logger.info(\"Detected Qwen3-8B variant\")\n            hidden_size = 4096\n            num_attention_heads = 32\n            num_kv_heads = 8\n            intermediate_size = 12288\n            head_dim = 128\n            max_position_embeddings = 40960\n        else:\n            # Unknown variant - try to detect from weights\n            logger.warning(\n                f\"Unknown Qwen3 variant: embed_hidden_size={embed_hidden_size}, layers={layer_count}. \"\n                \"Attempting to detect configuration from weights...\"\n            )\n            q_proj_weight = sd.get(\"model.layers.0.self_attn.q_proj.weight\")\n            k_proj_weight = sd.get(\"model.layers.0.self_attn.k_proj.weight\")\n            gate_proj_weight = sd.get(\"model.layers.0.mlp.gate_proj.weight\")\n\n            if q_proj_weight is None or k_proj_weight is None or gate_proj_weight is None:\n                raise ValueError(\"Could not find attention/mlp weights to determine configuration\")\n\n            hidden_size = embed_hidden_size\n            head_dim = 128\n            num_attention_heads = q_proj_weight.shape[0] // head_dim\n            num_kv_heads = k_proj_weight.shape[0] // head_dim\n            intermediate_size = gate_proj_weight.shape[0]\n            max_position_embeddings = 40960\n\n        logger.info(\n            f\"Qwen3 config: hidden_size={hidden_size}, layers={layer_count}, \"\n            f\"heads={num_attention_heads}, kv_heads={num_kv_heads}, intermediate={intermediate_size}\"\n        )\n\n        # Create Qwen3 config\n        qwen_config = Qwen3Config(\n            vocab_size=vocab_size,\n            hidden_size=hidden_size,\n            intermediate_size=intermediate_size,","sourceCodeStart":1051,"sourceCodeEnd":1087,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/model_manager/load/model_loaders/z_image.py#L1051-L1087","documentation":"After reading embed_tokens, the loader inspects layers.0's q_proj/k_proj (attention) and gate_proj (MLP) weights to infer head counts and intermediate size for the Qwen3 config. If none of these tensors exist under the expected names, the architecture cannot be detected and this ValueError is raised.","triggerScenarios":"The state dict has model.embed_tokens.weight but lacks model.layers.0.self_attn.q_proj.weight, k_proj.weight, or mlp.gate_proj.weight — e.g. layers stored with fused/packed attention names, a different layer indexing, or an export that dropped early-layer keys.","commonSituations":"Custom or quantized exports with fused QKV projections (qkv_proj) instead of separate q_proj/k_proj; checkpoints with a top-level prefix such as text_model.layers...; converted checkpoints where layer 0 was renumbered or pruned.","solutions":["Load a checkpoint with standard HF Qwen3 key naming (separate q_proj/k_proj/gate_proj per layer).","If keys are only prefixed, strip the prefix (e.g. text_model.) before/while loading so the expected names appear at the root.","If the checkpoint uses fused attention (qkv_proj), convert it to split q/k/v projections with a conversion script.","Dump sd.keys() and confirm model.layers.0.self_attn.q_proj.weight exists before loading."],"exampleFix":"// before\nsd = {\"model.layers.0.self_attn.qkv_proj.weight\": ...}  # fused keys -> error\n// after\nsd = {\"model.layers.0.self_attn.q_proj.weight\": ..., \"model.layers.0.self_attn.k_proj.weight\": ..., \"model.layers.0.mlp.gate_proj.weight\": ...}","handlingStrategy":"validation","validationCode":"REQUIRED = {\"model.layers.0.self_attn.q_proj.weight\",\n            \"model.layers.0.self_attn.k_proj.weight\",\n            \"model.layers.0.mlp.gate_proj.weight\"}\nwith safe_open(path, framework=\"pt\") as f:\n    missing = REQUIRED - set(f.keys())\nif missing:\n    raise ValueError(f\"Missing expected Qwen3 layer keys: {missing}\")","typeGuard":null,"tryCatchPattern":"try:\n    model = loader._load_model(config, submodel_type=SubModelType.TextEncoder)\nexcept ValueError as e:\n    if \"attention/mlp weights\" in str(e):\n        raise CheckpointFormatError(\"Fused/renamed layer keys; convert to HF Qwen3 naming\") from e\n    raise","preventionTips":["Avoid checkpoints with fused qkv_proj keys; convert to split q/k/v projections first.","Strip exporter prefixes (e.g. text_model.) so keys match HF Qwen3 names.","Dump state-dict keys and diff against a known-good Qwen3 checkpoint before loading.","Pin to officially supported Z-Image text-encoder exports."],"tags":["invokeai","z-image","qwen3","checkpoint","missing-key","config-detection"],"backgroundTag":"checkpoint-missing-weight-key","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}