{"record":{"id":"616f628869d7ad69","repo":"huggingface/candle","slug":"query-n-heads-must-be-a-multiple-of-n-kv-heads","errorCode":null,"errorMessage":"query `n_heads` must be a multiple of `n_kv_heads`","messagePattern":"query `n_heads` must be a multiple of `n_kv_heads`","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":1070,"sourceCode":"        let output = device\n            .new_buffer_builder()\n            .with_size_for(elem_count, q.dtype())\n            .with_label(\"sdpa_o\")\n            .build()?;\n\n        // q,k must have matching emb dim\n        if q_l.dim(D::Minus1)? != k_l.dim(D::Minus1)? {\n            candle::bail!(\"`q` and `k` last dims must match\");\n        }\n\n        // k,v must have matching n kv heads\n        if v_l.dim(D::Minus(3))? != k_l.dim(D::Minus(3))? {\n            candle::bail!(\"`k` and `v` head dims must match\");\n        }\n\n        // n_heads % n_kv_heads == 0; n_heads >= 1, n_kv_heads >= 1.\n        if q_l.dim(D::Minus(3))? % k_l.dim(D::Minus(3))? != 0 {\n            candle::bail!(\"query `n_heads` must be a multiple of `n_kv_heads`\");\n        }\n\n        let k_head = k_l.dim(D::Minus1)?;\n        let q_head = q_l.dim(D::Minus1)?;\n        let q_seq = q_l.dim(2)?;\n        let k_seq = k_l.dim(2)?;\n\n        let mut implementation_supports_use_case = q_head == k_head;\n        let supported_head_dim = q_head == 32\n            || q_head == 64\n            || q_head == 72\n            || q_head == 80\n            || q_head == 96\n            || q_head == 128\n            || q_head == 256\n            || q_head == 512;\n\n        let supports_sdpa_full_mask = self.mask.is_none() || q_seq <= k_seq;","sourceCodeStart":1052,"sourceCodeEnd":1088,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L1052-L1088","documentation":"The Metal SDPA kernel implements grouped-query attention only when the number of query heads is an exact multiple of the number of KV heads (n_heads % n_kv_heads == 0), so heads can be evenly distributed across KV groups. Non-divisible configurations are rejected.","triggerScenarios":"Calling SDPA on Metal with q head count (dim D::Minus(3)) not divisible by k head count, e.g. q with 10 heads and k/v with 4 KV heads.","commonSituations":"Hand-rolled GQA configs with arbitrary head counts; off-by-one in head splitting; porting models where num_attention_heads is not a multiple of num_key_value_heads; typos in config files.","solutions":["Choose num_attention_heads as a multiple of num_key_value_heads (e.g. 32/8 or 32/4)","Adjust the model config so q_heads % kv_heads == 0","If the model truly has incompatible head counts, fall back to explicit repeat_interleave + standard matmul attention instead of the fused SDPA","Add a config-time assertion so the invalid pairing fails at load, not at runtime"],"exampleFix":"// before\nnum_attention_heads: 10, num_key_value_heads: 4 // 10 % 4 != 0\n// after\nnum_attention_heads: 12, num_key_value_heads: 4 // 12 % 4 == 0","handlingStrategy":"validation","validationCode":"fn check_gqa_heads(q: &Tensor, k: &Tensor) -> candle::Result<()> {\n    let nq = q.dim(candle::D::Minus(3))?;\n    let nkv = k.dim(candle::D::Minus(3))?;\n    if nq % nkv != 0 {\n        candle::bail!(\"n_heads {nq} not a multiple of n_kv_heads {nkv}\");\n    }\n    Ok(())\n}","typeGuard":"fn gqa_config_ok(n_heads: usize, n_kv_heads: usize) -> bool {\n    n_heads >= 1 && n_kv_heads >= 1 && n_heads % n_kv_heads == 0\n}","tryCatchPattern":"match sdpa(&q, &k, &v, &mask, false, Some(scale)) {\n    Ok(y) => y,\n    Err(e) if e.to_string().contains(\"multiple of\") => repeat_kv_manual_attention(&q, &k, &v, scale)?,\n    Err(e) => Err(e),\n}","preventionTips":["Validate at config load: num_attention_heads % num_key_value_heads == 0","Use standard GQA ratios (e.g. 32/8, 32/4)","Fail fast at model construction, not at first forward","If head counts are fixed by a checkpoint, derive one from the other"],"tags":["sdpa","gqa","metal","attention"],"backgroundTag":"attention-head-count-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}