{"record":{"id":"6644fbef75a7e6ea","repo":"huggingface/candle","slug":"q-and-k-last-dims-must-match","errorCode":null,"errorMessage":"`q` and `k` last dims must match","messagePattern":"`q` and `k` last dims must match","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":1060,"sourceCode":"        use candle::backend::BackendStorage;\n        use candle_metal_kernels::SdpaDType;\n\n        let device = q.device();\n\n        let out_dims = vec![q_l.dim(0)?, q_l.dim(1)?, q_l.dim(2)?, v_l.dim(3)?];\n        let elem_count: usize = out_dims.iter().product();\n        let out_shape = Shape::from_dims(&out_dims);\n        let out_layout = Layout::contiguous(out_shape.clone());\n\n        let output = device\n            .new_buffer_builder()\n            .with_size_for(elem_count, q.dtype())\n            .with_label(\"sdpa_o\")\n            .build()?;\n\n        // q,k must have matching emb dim\n        if q_l.dim(D::Minus1)? != k_l.dim(D::Minus1)? {\n            candle::bail!(\"`q` and `k` last dims must match\");\n        }\n\n        // k,v must have matching n kv heads\n        if v_l.dim(D::Minus(3))? != k_l.dim(D::Minus(3))? {\n            candle::bail!(\"`k` and `v` head dims must match\");\n        }\n\n        // n_heads % n_kv_heads == 0; n_heads >= 1, n_kv_heads >= 1.\n        if q_l.dim(D::Minus(3))? % k_l.dim(D::Minus(3))? != 0 {\n            candle::bail!(\"query `n_heads` must be a multiple of `n_kv_heads`\");\n        }\n\n        let k_head = k_l.dim(D::Minus1)?;\n        let q_head = q_l.dim(D::Minus1)?;\n        let q_seq = q_l.dim(2)?;\n        let k_seq = k_l.dim(2)?;\n\n        let mut implementation_supports_use_case = q_head == k_head;","sourceCodeStart":1042,"sourceCodeEnd":1078,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L1042-L1078","documentation":"In the Metal SDPA kernel, the query and key tensors must agree in their last (head/embedding) dimension so Q·K^T is a valid matmul per head. The op checks q_l.dim(D::Minus1) == k_l.dim(D::Minus1) and bails otherwise.","triggerScenarios":"Calling the SDPA op on Metal with q and k tensors whose last dims differ, e.g. mismatched head_dim between query and key projections, or k carrying a different embedding size.","commonSituations":"Misconfigured GQA/model projections where q_proj outputs a different head dim than k_proj; reshaping errors that flatten wrong dims; feeding cross-attention with mismatched encoder/decoder dims without a projection.","solutions":["Make q and k projections produce the same head_dim (adjust linear layer out_features)","Fix the reshape/split so both q and k end with head_dim as the last dimension","Add a projection layer (linear) to map one side's head dim to the other's","Validate q.dims()[..].last() == k.dims()[..].last() before calling SDPA"],"exampleFix":"// before\nlet q = q_proj.forward(&x)?;   // head_dim 128\nlet k = k_proj_small.forward(&x)?; // head_dim 64\nlet out = sdpa(&q, &k, &v, ...)?;\n// after\nassert_eq!(q.dim(D::Minus1)?, k.dim(D::Minus1)?);\nlet k = k_proj.forward(&x)?; // head_dim 128\nlet out = sdpa(&q, &k, &v, ...)?;","handlingStrategy":"validation","validationCode":"fn check_qk_last_dim(q: &Tensor, k: &Tensor) -> candle::Result<()> {\n    if q.dim(candle::D::Minus1)? != k.dim(candle::D::Minus1)? {\n        candle::bail!(\"q head_dim {:?} != k head_dim {:?}\", q.shape(), k.shape());\n    }\n    Ok(())\n}","typeGuard":"fn qk_dims_ok(q: &Tensor, k: &Tensor) -> bool {\n    q.dim(candle::D::Minus1) == k.dim(candle::D::Minus1) && q.dim(candle::D::Minus1).is_ok()\n}","tryCatchPattern":"match sdpa(&q, &k, &v, &mask, false, Some(scale)) {\n    Ok(y) => y,\n    Err(e) if e.to_string().contains(\"last dims must match\") => Err(candle::Error::msg(\"q/k head_dim config bug\").bt()),\n    Err(e) => Err(e),\n}","preventionTips":["Ensure q_proj and k_proj output the same head_dim","Assert head_dim equality when reshaping projections into heads","For cross-attention, project keys to the query head dim","Check reshapes keep head_dim as the last dimension"],"tags":["sdpa","shape-mismatch","metal","attention"],"backgroundTag":"attention-shape-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}