What causes "Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}." in huggingface/candle?

Ensure q_seq == 1 implies k_seq >= 1 (fix KV cache length bookkeeping)

Meta SDPA does not support q dims {:?}, k dims {:?}, v dims

Error message

Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.

What it means

Beyond head-dim support, the Metal SDPA implementation only handles two use cases: full attention (q_seq > 1 with supported head dim, no incompatible mask, supported dtype) and vector attention (q_seq == 1 with supported head dim and q_seq <= k_seq). Anything else — e.g. a mask requiring more support, q_seq > k_seq with a single query token, head_dim 512 in F32 — fails this check with all dims printed.

Source

Thrown at candle-nn/src/ops.rs:1106

        let supports_sdpa_full_mask = self.mask.is_none() || q_seq <= k_seq;
        // F32 full attention at head_dim=512 exceeds 32KB Metal threadgroup memory
        let supports_sdpa_full_dtype = !(q_head == 512 && q.dtype() == DType::F32);
        let supports_sdpa_full =
            q_seq > 1 && supported_head_dim && supports_sdpa_full_mask && supports_sdpa_full_dtype;
        let supports_sdpa_vector = q_seq == 1 && supported_head_dim && q_seq <= k_seq;

        implementation_supports_use_case &= supports_sdpa_full || supports_sdpa_vector;

        if !supported_head_dim {
            candle::bail!(
                "Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.",
                q_l.dims(),
                k_l.dims(),
                v_l.dims()
            );
        }
        if !implementation_supports_use_case {
            candle::bail!(
                "Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.",
                q_l.dims(),
                k_l.dims(),
                v_l.dims()
            );
        }

        for t in [k.dtype(), v.dtype()] {
            if q.dtype() != t {
                candle::bail!("all q, k, v dtypes must match.");
            }
        }

        let itype = match q.dtype() {
            DType::BF16 => SdpaDType::BF16,
            DType::F16 => SdpaDType::F16,
            DType::F32 => SdpaDType::F32,
            other => candle::bail!("unsupported sdpa type {other:?}"),

View on GitHub (pinned to d5fee525bf)

Solutions

Ensure q_seq == 1 implies k_seq >= 1 (fix KV cache length bookkeeping)
Drop or reshape the attention mask so it's compatible with the full kernel, or apply it manually after SDPA
Use F16/BF16 instead of F32 when head_dim == 512
Fall back to manual matmul-based attention for unsupported seq/mask combinations

Example fix

// before
let out = sdpa(&q, &k, &v, Some(&mask), false, Some(1.0))?; // mask unsupported at q_seq>1
// after
let attn = (q.matmul(&k.t()?)? * scale)?.softmax(D::Minus1)?;
let attn = apply_mask(&attn, &mask)?;
let out = attn.matmul(&v)?;

Defensive patterns

Strategy: fallback

Validate before calling

fn sdpa_use_case_ok(q_seq: usize, k_seq: usize, mask: Option<&Tensor>, head_dim: usize, dt: candle::DType) -> bool {
    let supported_head_dim = matches!(head_dim, 32|64|72|80|96|128|256|512);
    let full = q_seq > 1 && supported_head_dim && (mask.is_none() || q_seq <= k_seq) && !(head_dim == 512 && dt == candle::DType::F32);
    let vector = q_seq == 1 && supported_head_dim && q_seq <= k_seq;
    full || vector
}

Try / catch

let out = match sdpa(&q, &k, &v, mask, do_causal, Some(scale)) {
    Ok(y) => y,
    Err(e) if e.to_string().contains("Meta SDPA does not support") => manual_attention_with_mask(&q, &k, &v, mask, scale)?,
    Err(e) => return Err(e),
};

Prevention

Fix KV-cache length bookkeeping so q_seq <= k_seq always holds for single-token decode
Apply exotic masks manually outside the fused kernel
Avoid F32 at head_dim 512
Keep a matmul+softmax attention fallback for Metal edge cases

When it happens

Trigger: Calling SDPA on Metal with q_seq == 1 but k_seq < q_seq (impossible), q_seq > 1 with a mask not compatible with the full kernel, or head_dim 512 with F32 dtype (threadgroup memory limit).

Common situations: Single-token decode where the KV cache was trimmed below one entry; custom attention masks in generation loops; extreme head_dim models in F32; chunked prefill patterns the kernel doesn't cover.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/96ca1143d1c59e60. Report an issue: GitHub.