huggingface/candle · error

query `n_heads` must be a multiple of `n_kv_heads`

Error message

query `n_heads` must be a multiple of `n_kv_heads`

What it means

The Metal SDPA kernel implements grouped-query attention only when the number of query heads is an exact multiple of the number of KV heads (n_heads % n_kv_heads == 0), so heads can be evenly distributed across KV groups. Non-divisible configurations are rejected.

Source

Thrown at candle-nn/src/ops.rs:1070

        let output = device
            .new_buffer_builder()
            .with_size_for(elem_count, q.dtype())
            .with_label("sdpa_o")
            .build()?;

        // q,k must have matching emb dim
        if q_l.dim(D::Minus1)? != k_l.dim(D::Minus1)? {
            candle::bail!("`q` and `k` last dims must match");
        }

        // k,v must have matching n kv heads
        if v_l.dim(D::Minus(3))? != k_l.dim(D::Minus(3))? {
            candle::bail!("`k` and `v` head dims must match");
        }

        // n_heads % n_kv_heads == 0; n_heads >= 1, n_kv_heads >= 1.
        if q_l.dim(D::Minus(3))? % k_l.dim(D::Minus(3))? != 0 {
            candle::bail!("query `n_heads` must be a multiple of `n_kv_heads`");
        }

        let k_head = k_l.dim(D::Minus1)?;
        let q_head = q_l.dim(D::Minus1)?;
        let q_seq = q_l.dim(2)?;
        let k_seq = k_l.dim(2)?;

        let mut implementation_supports_use_case = q_head == k_head;
        let supported_head_dim = q_head == 32
            || q_head == 64
            || q_head == 72
            || q_head == 80
            || q_head == 96
            || q_head == 128
            || q_head == 256
            || q_head == 512;

        let supports_sdpa_full_mask = self.mask.is_none() || q_seq <= k_seq;

View on GitHub (pinned to d5fee525bf)

Solutions

  1. Choose num_attention_heads as a multiple of num_key_value_heads (e.g. 32/8 or 32/4)
  2. Adjust the model config so q_heads % kv_heads == 0
  3. If the model truly has incompatible head counts, fall back to explicit repeat_interleave + standard matmul attention instead of the fused SDPA
  4. Add a config-time assertion so the invalid pairing fails at load, not at runtime

Example fix

// before
num_attention_heads: 10, num_key_value_heads: 4 // 10 % 4 != 0
// after
num_attention_heads: 12, num_key_value_heads: 4 // 12 % 4 == 0
Defensive patterns

Strategy: validation

Validate before calling

fn check_gqa_heads(q: &Tensor, k: &Tensor) -> candle::Result<()> {
    let nq = q.dim(candle::D::Minus(3))?;
    let nkv = k.dim(candle::D::Minus(3))?;
    if nq % nkv != 0 {
        candle::bail!("n_heads {nq} not a multiple of n_kv_heads {nkv}");
    }
    Ok(())
}

Type guard

fn gqa_config_ok(n_heads: usize, n_kv_heads: usize) -> bool {
    n_heads >= 1 && n_kv_heads >= 1 && n_heads % n_kv_heads == 0
}

Try / catch

match sdpa(&q, &k, &v, &mask, false, Some(scale)) {
    Ok(y) => y,
    Err(e) if e.to_string().contains("multiple of") => repeat_kv_manual_attention(&q, &k, &v, scale)?,
    Err(e) => Err(e),
}

Prevention

When it happens

Trigger: Calling SDPA on Metal with q head count (dim D::Minus(3)) not divisible by k head count, e.g. q with 10 heads and k/v with 4 KV heads.

Common situations: Hand-rolled GQA configs with arbitrary head counts; off-by-one in head splitting; porting models where num_attention_heads is not a multiple of num_key_value_heads; typos in config files.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/616f628869d7ad69. Report an issue: GitHub.