huggingface/candle · error

Meta SDPA does not support q head dim {q_head}: q dims {:?},

Error message

Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.

What it means

The Metal SDPA kernels are pre-compiled for a fixed set of head dimensions (32, 64, 72, 80, 96, 128, 256, 512). If the query head dim is outside this list, no suitable Metal kernel exists and the op bails, printing the full q/k/v dims.

Source

Thrown at candle-nn/src/ops.rs:1098

            || q_head == 64
            || q_head == 72
            || q_head == 80
            || q_head == 96
            || q_head == 128
            || q_head == 256
            || q_head == 512;

        let supports_sdpa_full_mask = self.mask.is_none() || q_seq <= k_seq;
        // F32 full attention at head_dim=512 exceeds 32KB Metal threadgroup memory
        let supports_sdpa_full_dtype = !(q_head == 512 && q.dtype() == DType::F32);
        let supports_sdpa_full =
            q_seq > 1 && supported_head_dim && supports_sdpa_full_mask && supports_sdpa_full_dtype;
        let supports_sdpa_vector = q_seq == 1 && supported_head_dim && q_seq <= k_seq;

        implementation_supports_use_case &= supports_sdpa_full || supports_sdpa_vector;

        if !supported_head_dim {
            candle::bail!(
                "Meta SDPA does not support q head dim {q_head}: q dims {:?}, k dims {:?}, v dims {:?}.",
                q_l.dims(),
                k_l.dims(),
                v_l.dims()
            );
        }
        if !implementation_supports_use_case {
            candle::bail!(
                "Meta SDPA does not support q dims {:?}, k dims {:?}, v dims {:?}.",
                q_l.dims(),
                k_l.dims(),
                v_l.dims()
            );
        }

        for t in [k.dtype(), v.dtype()] {
            if q.dtype() != t {
                candle::bail!("all q, k, v dtypes must match.");

View on GitHub (pinned to d5fee525bf)

Solutions

1. Pick hidden_size and n_heads so head_dim is one of the supported values (32, 64, 72, 80, 96, 128, 256, 512)
2. Pad head_dim to a supported size (with matching zero padding on outputs) or project q/k/v to a supported head dim
3. Fall back to manual attention (matmul + softmax) for non-supported head dims on Metal
4. Also avoid head_dim 512 with F32, which is excluded from the full-attention path

Example fix

// before
let head_dim = hidden_size / n_heads; // e.g. 40
let out = sdpa(&q, &k, &v, ...)?;
// after
assert!([32,64,72,80,96,128,256,512].contains(&head_dim), "unsupported head dim {head_dim}");
let out = sdpa(&q, &k, &v, ...)?;

const SUPPORTED_HEAD_DIMS: [usize; 8] = [32, 64, 72, 80, 96, 128, 256, 512];
fn head_dim_supported(q: &Tensor) -> bool {
    q.dim(candle::D::Minus1).map(|d| SUPPORTED_HEAD_DIMS.contains(&d)).unwrap_or(false)
}

fn supported_sdpa_head_dim(d: usize) -> bool {
    matches!(d, 32 | 64 | 72 | 80 | 96 | 128 | 256 | 512)
}

let out = if head_dim_supported(&q) {
    sdpa(&q, &k, &v, &mask, false, Some(scale))?
} else {
    manual_attention(&q, &k, &v, scale)?
};

When it happens

Trigger: Calling SDPA on Metal with a q head dim (last dim) not in {32,64,72,80,96,128,256,512}, e.g. head_dim 40 from hidden_size 320 / 8 heads, or odd hidden sizes in custom models.

Common situations: Custom transformer models with non-standard head sizes; older GPT-style dims like 96 is fine but 40/56 are not; quantization or projection changes that alter head_dim.

Related errors

• Meta SDPA does not support q dims {:?}, k dims {:?}, v dims
• Metal device does not yet support F8E4M3.
• `q` and `k` last dims must match
• `k` and `v` head dims must match
• query `n_heads` must be a multiple of `n_kv_heads`

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/3bc8499f59d42b7c. Report an issue: GitHub.