{"record":{"id":"3bc8499f59d42b7c","repo":"huggingface/candle","slug":"meta-sdpa-does-not-support-q-head-dim-q-head-q","errorCode":null,"errorMessage":"Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.","messagePattern":"Meta SDPA does not support q head dim (.+?): q dims (.+?), k dims (.+?), v dims (.+?)\\.","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":1098,"sourceCode":"            || q_head == 64\n            || q_head == 72\n            || q_head == 80\n            || q_head == 96\n            || q_head == 128\n            || q_head == 256\n            || q_head == 512;\n\n        let supports_sdpa_full_mask = self.mask.is_none() || q_seq <= k_seq;\n        // F32 full attention at head_dim=512 exceeds 32KB Metal threadgroup memory\n        let supports_sdpa_full_dtype = !(q_head == 512 && q.dtype() == DType::F32);\n        let supports_sdpa_full =\n            q_seq > 1 && supported_head_dim && supports_sdpa_full_mask && supports_sdpa_full_dtype;\n        let supports_sdpa_vector = q_seq == 1 && supported_head_dim && q_seq <= k_seq;\n\n        implementation_supports_use_case &= supports_sdpa_full || supports_sdpa_vector;\n\n        if !supported_head_dim {\n            candle::bail!(\n                \"Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.\",\n                q_l.dims(),\n                k_l.dims(),\n                v_l.dims()\n            );\n        }\n        if !implementation_supports_use_case {\n            candle::bail!(\n                \"Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.\",\n                q_l.dims(),\n                k_l.dims(),\n                v_l.dims()\n            );\n        }\n\n        for t in [k.dtype(), v.dtype()] {\n            if q.dtype() != t {\n                candle::bail!(\"all q, k, v dtypes must match.\");","sourceCodeStart":1080,"sourceCodeEnd":1116,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L1080-L1116","documentation":"The Metal SDPA kernels are pre-compiled for a fixed set of head dimensions (32, 64, 72, 80, 96, 128, 256, 512). If the query head dim is outside this list, no suitable Metal kernel exists and the op bails, printing the full q/k/v dims.","triggerScenarios":"Calling SDPA on Metal with a q head dim (last dim) not in {32,64,72,80,96,128,256,512}, e.g. head_dim 40 from hidden_size 320 / 8 heads, or odd hidden sizes in custom models.","commonSituations":"Custom transformer models with non-standard head sizes; older GPT-style dims like 96 is fine but 40/56 are not; quantization or projection changes that alter head_dim.","solutions":["Pick hidden_size and n_heads so head_dim is one of the supported values (32, 64, 72, 80, 96, 128, 256, 512)","Pad head_dim to a supported size (with matching zero padding on outputs) or project q/k/v to a supported head dim","Fall back to manual attention (matmul + softmax) for non-supported head dims on Metal","Also avoid head_dim 512 with F32, which is excluded from the full-attention path"],"exampleFix":"// before\nlet head_dim = hidden_size / n_heads; // e.g. 40\nlet out = sdpa(&q, &k, &v, ...)?;\n// after\nassert!([32,64,72,80,96,128,256,512].contains(&head_dim), \"unsupported head dim {head_dim}\");\nlet out = sdpa(&q, &k, &v, ...)?;","handlingStrategy":"validation","validationCode":"const SUPPORTED_HEAD_DIMS: [usize; 8] = [32, 64, 72, 80, 96, 128, 256, 512];\nfn head_dim_supported(q: &Tensor) -> bool {\n    q.dim(candle::D::Minus1).map(|d| SUPPORTED_HEAD_DIMS.contains(&d)).unwrap_or(false)\n}","typeGuard":"fn supported_sdpa_head_dim(d: usize) -> bool {\n    matches!(d, 32 | 64 | 72 | 80 | 96 | 128 | 256 | 512)\n}","tryCatchPattern":"let out = if head_dim_supported(&q) {\n    sdpa(&q, &k, &v, &mask, false, Some(scale))?\n} else {\n    manual_attention(&q, &k, &v, scale)?\n};","preventionTips":["Choose hidden_size/n_heads so head_dim lands on a supported value","Assert supported head dims at model build time","Avoid F32 with head_dim 512 (also unsupported for full attention)","Check the metal kernels list of supported dims when upgrading candle"],"tags":["sdpa","metal","head-dim","unsupported"],"backgroundTag":"unsupported-head-dimension","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}