{"record":{"id":"88b9e569e3344027","repo":"huggingface/candle","slug":"all-q-k-v-dtypes-must-match","errorCode":null,"errorMessage":"all q, k, v dtypes must match.","messagePattern":"all q, k, v dtypes must match\\.","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":1116,"sourceCode":"            candle::bail!(\n                \"Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.\",\n                q_l.dims(),\n                k_l.dims(),\n                v_l.dims()\n            );\n        }\n        if !implementation_supports_use_case {\n            candle::bail!(\n                \"Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.\",\n                q_l.dims(),\n                k_l.dims(),\n                v_l.dims()\n            );\n        }\n\n        for t in [k.dtype(), v.dtype()] {\n            if q.dtype() != t {\n                candle::bail!(\"all q, k, v dtypes must match.\");\n            }\n        }\n\n        let itype = match q.dtype() {\n            DType::BF16 => SdpaDType::BF16,\n            DType::F16 => SdpaDType::F16,\n            DType::F32 => SdpaDType::F32,\n            other => candle::bail!(\"unsupported sdpa type {other:?}\"),\n        };\n\n        let encoder = q.device().command_encoder()?;\n        if supports_sdpa_vector {\n            // Route to the 2 pass fused attention if the k seqlen is large.\n            // https://github.com/ml-explore/mlx/pull/1597\n            const TWO_PASS_K_THRESHOLD: usize = 1024;\n            if k_seq >= TWO_PASS_K_THRESHOLD {\n                let mut intermediate_shape = [\n                    &out_dims[0..out_dims.len() - 2],","sourceCodeStart":1098,"sourceCodeEnd":1134,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L1098-L1134","documentation":"The Metal SDPA kernel requires the query, key, and value tensors to share one dtype; it selects a single SdpaDType (BF16/F16/F32) from q and runs one fused pipeline over all three inputs. Mixed-dtype inputs (e.g. F16 cache with BF16 query) are rejected.","triggerScenarios":"Calling SDPA on Metal where q.dtype() != k.dtype() or != v.dtype(), commonly when the KV cache was written in a different dtype than the query, or when a projection layer outputs a different dtype than cached tensors.","commonSituations":"Quantized KV caches (e.g. F16 cache with BF16 activations); dtype changes mid-model after loading mixed checkpoints; forgetting .to_dtype on one of q/k/v after a device/dtype migration.","solutions":["Convert q, k, v to the same dtype with Tensor::to_dtype before calling SDPA","Configure the KV cache dtype to match the model activation dtype","Fix checkpoint loading so all projections produce the model dtype","Validate q.dtype() == k.dtype() && q.dtype() == v.dtype() before the call"],"exampleFix":"// before\nlet out = sdpa(&q, &k.to_dtype(DType::F32)?, &v, ...)?; // mixed\n// after\nlet dt = q.dtype();\nlet out = sdpa(&q, &k.to_dtype(dt)?, &v.to_dtype(dt)?, ...)?;","handlingStrategy":"validation","validationCode":"fn check_sdpa_dtypes(q: &Tensor, k: &Tensor, v: &Tensor) -> candle::Result<()> {\n    let dt = q.dtype();\n    if k.dtype() != dt || v.dtype() != dt {\n        candle::bail!(\"sdpa dtype mismatch: q={dt:?} k={:?} v={:?}\", k.dtype(), v.dtype());\n    }\n    Ok(())\n}","typeGuard":"fn sdpa_dtypes_ok(q: &Tensor, k: &Tensor, v: &Tensor) -> bool {\n    q.dtype() == k.dtype() && q.dtype() == v.dtype()\n}","tryCatchPattern":"let out = if sdpa_dtypes_ok(&q, &k, &v) {\n    sdpa(&q, &k, &v, &mask, false, Some(scale))?\n} else {\n    let dt = q.dtype();\n    sdpa(&q, &k.to_dtype(dt)?, &v.to_dtype(dt)?, &mask, false, Some(scale))?\n};","preventionTips":["Match KV cache dtype to activation dtype at cache creation","Call .to_dtype on all of q/k/v when mixing checkpoints or caches","Standardize the model on one dtype (F16/BF16/F32) end to end","Check dtypes once at model load and convert weights then"],"tags":["sdpa","dtype-mismatch","metal","candle"],"backgroundTag":"dtype-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}