huggingface/candle · error

all q, k, v dtypes must match.

Error message

all q, k, v dtypes must match.

What it means

The Metal SDPA kernel requires the query, key, and value tensors to share one dtype; it selects a single SdpaDType (BF16/F16/F32) from q and runs one fused pipeline over all three inputs. Mixed-dtype inputs (e.g. F16 cache with BF16 query) are rejected.

Source

Thrown at candle-nn/src/ops.rs:1116

            candle::bail!(
                "Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.",
                q_l.dims(),
                k_l.dims(),
                v_l.dims()
            );
        }
        if !implementation_supports_use_case {
            candle::bail!(
                "Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.",
                q_l.dims(),
                k_l.dims(),
                v_l.dims()
            );
        }

        for t in [k.dtype(), v.dtype()] {
            if q.dtype() != t {
                candle::bail!("all q, k, v dtypes must match.");
            }
        }

        let itype = match q.dtype() {
            DType::BF16 => SdpaDType::BF16,
            DType::F16 => SdpaDType::F16,
            DType::F32 => SdpaDType::F32,
            other => candle::bail!("unsupported sdpa type {other:?}"),
        };

        let encoder = q.device().command_encoder()?;
        if supports_sdpa_vector {
            // Route to the 2 pass fused attention if the k seqlen is large.
            // https://github.com/ml-explore/mlx/pull/1597
            const TWO_PASS_K_THRESHOLD: usize = 1024;
            if k_seq >= TWO_PASS_K_THRESHOLD {
                let mut intermediate_shape = [
                    &out_dims[0..out_dims.len() - 2],

View on GitHub (pinned to d5fee525bf)

Solutions

1. Convert q, k, v to the same dtype with Tensor::to_dtype before calling SDPA
2. Configure the KV cache dtype to match the model activation dtype
3. Fix checkpoint loading so all projections produce the model dtype
4. Validate q.dtype() == k.dtype() && q.dtype() == v.dtype() before the call

Example fix

// before
let out = sdpa(&q, &k.to_dtype(DType::F32)?, &v, ...)?; // mixed
// after
let dt = q.dtype();
let out = sdpa(&q, &k.to_dtype(dt)?, &v.to_dtype(dt)?, ...)?;

fn check_sdpa_dtypes(q: &Tensor, k: &Tensor, v: &Tensor) -> candle::Result<()> {
    let dt = q.dtype();
    if k.dtype() != dt || v.dtype() != dt {
        candle::bail!("sdpa dtype mismatch: q={dt:?} k={:?} v={:?}", k.dtype(), v.dtype());
    }
    Ok(())
}

fn sdpa_dtypes_ok(q: &Tensor, k: &Tensor, v: &Tensor) -> bool {
    q.dtype() == k.dtype() && q.dtype() == v.dtype()
}

let out = if sdpa_dtypes_ok(&q, &k, &v) {
    sdpa(&q, &k, &v, &mask, false, Some(scale))?
} else {
    let dt = q.dtype();
    sdpa(&q, &k.to_dtype(dt)?, &v.to_dtype(dt)?, &mask, false, Some(scale))?
};

When it happens

Trigger: Calling SDPA on Metal where q.dtype() != k.dtype() or != v.dtype(), commonly when the KV cache was written in a different dtype than the query, or when a projection layer outputs a different dtype than cached tensors.

Common situations: Quantized KV caches (e.g. F16 cache with BF16 activations); dtype changes mid-model after loading mixed checkpoints; forgetting .to_dtype on one of q/k/v after a device/dtype migration.

Related errors

• dtype mismatch, expected {:?}, got {:?}
• Invalid where: different dtypes for values {:?} != {:?}
• unsupported const-set f8e4m3
• unsupported const-set f64
• Metal contiguous to_dtype {left:?} {right:?} not implemented

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/88b9e569e3344027. Report an issue: GitHub.