{"record":{"id":"96ca1143d1c59e60","repo":"huggingface/candle","slug":"meta-sdpa-does-not-support-q-dims-k-dims","errorCode":null,"errorMessage":"Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.","messagePattern":"Meta SDPA does not support q dims (.+?), k dims (.+?), v dims (.+?)\\.","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":1106,"sourceCode":"        let supports_sdpa_full_mask = self.mask.is_none() || q_seq <= k_seq;\n        // F32 full attention at head_dim=512 exceeds 32KB Metal threadgroup memory\n        let supports_sdpa_full_dtype = !(q_head == 512 && q.dtype() == DType::F32);\n        let supports_sdpa_full =\n            q_seq > 1 && supported_head_dim && supports_sdpa_full_mask && supports_sdpa_full_dtype;\n        let supports_sdpa_vector = q_seq == 1 && supported_head_dim && q_seq <= k_seq;\n\n        implementation_supports_use_case &= supports_sdpa_full || supports_sdpa_vector;\n\n        if !supported_head_dim {\n            candle::bail!(\n                \"Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.\",\n                q_l.dims(),\n                k_l.dims(),\n                v_l.dims()\n            );\n        }\n        if !implementation_supports_use_case {\n            candle::bail!(\n                \"Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.\",\n                q_l.dims(),\n                k_l.dims(),\n                v_l.dims()\n            );\n        }\n\n        for t in [k.dtype(), v.dtype()] {\n            if q.dtype() != t {\n                candle::bail!(\"all q, k, v dtypes must match.\");\n            }\n        }\n\n        let itype = match q.dtype() {\n            DType::BF16 => SdpaDType::BF16,\n            DType::F16 => SdpaDType::F16,\n            DType::F32 => SdpaDType::F32,\n            other => candle::bail!(\"unsupported sdpa type {other:?}\"),","sourceCodeStart":1088,"sourceCodeEnd":1124,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L1088-L1124","documentation":"Beyond head-dim support, the Metal SDPA implementation only handles two use cases: full attention (q_seq > 1 with supported head dim, no incompatible mask, supported dtype) and vector attention (q_seq == 1 with supported head dim and q_seq <= k_seq). Anything else — e.g. a mask requiring more support, q_seq > k_seq with a single query token, head_dim 512 in F32 — fails this check with all dims printed.","triggerScenarios":"Calling SDPA on Metal with q_seq == 1 but k_seq < q_seq (impossible), q_seq > 1 with a mask not compatible with the full kernel, or head_dim 512 with F32 dtype (threadgroup memory limit).","commonSituations":"Single-token decode where the KV cache was trimmed below one entry; custom attention masks in generation loops; extreme head_dim models in F32; chunked prefill patterns the kernel doesn't cover.","solutions":["Ensure q_seq == 1 implies k_seq >= 1 (fix KV cache length bookkeeping)","Drop or reshape the attention mask so it's compatible with the full kernel, or apply it manually after SDPA","Use F16/BF16 instead of F32 when head_dim == 512","Fall back to manual matmul-based attention for unsupported seq/mask combinations"],"exampleFix":"// before\nlet out = sdpa(&q, &k, &v, Some(&mask), false, Some(1.0))?; // mask unsupported at q_seq>1\n// after\nlet attn = (q.matmul(&k.t()?)? * scale)?.softmax(D::Minus1)?;\nlet attn = apply_mask(&attn, &mask)?;\nlet out = attn.matmul(&v)?;","handlingStrategy":"fallback","validationCode":"fn sdpa_use_case_ok(q_seq: usize, k_seq: usize, mask: Option<&Tensor>, head_dim: usize, dt: candle::DType) -> bool {\n    let supported_head_dim = matches!(head_dim, 32|64|72|80|96|128|256|512);\n    let full = q_seq > 1 && supported_head_dim && (mask.is_none() || q_seq <= k_seq) && !(head_dim == 512 && dt == candle::DType::F32);\n    let vector = q_seq == 1 && supported_head_dim && q_seq <= k_seq;\n    full || vector\n}","typeGuard":null,"tryCatchPattern":"let out = match sdpa(&q, &k, &v, mask, do_causal, Some(scale)) {\n    Ok(y) => y,\n    Err(e) if e.to_string().contains(\"Meta SDPA does not support\") => manual_attention_with_mask(&q, &k, &v, mask, scale)?,\n    Err(e) => return Err(e),\n};","preventionTips":["Fix KV-cache length bookkeeping so q_seq <= k_seq always holds for single-token decode","Apply exotic masks manually outside the fused kernel","Avoid F32 at head_dim 512","Keep a matmul+softmax attention fallback for Metal edge cases"],"tags":["sdpa","metal","mask","unsupported"],"backgroundTag":"unsupported-attention-configuration","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}