{"record":{"id":"73751e210631c0b2","repo":"sgl-project/sglang","slug":"gqa-mqa-requires-query-heads-to-be-a-multiple-of-k","errorCode":null,"errorMessage":"GQA/MQA requires query heads to be a multiple of KV heads, got q_heads={query.shape[1]} and kv_heads={key.shape[1]}","messagePattern":"GQA/MQA requires query heads to be a multiple of KV heads, got q_heads=(.+?) and kv_heads=(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/sage_attn3.py","lineNumber":71,"sourceCode":"        self.causal = causal\n        self.softmax_scale = softmax_scale\n        self.dropout = extra_impl_args.get(\"dropout_p\", 0.0)\n\n    def forward(\n        self,\n        query: torch.Tensor,\n        key: torch.Tensor,\n        value: torch.Tensor,\n        attn_metadata: AttentionMetadata,\n    ) -> torch.Tensor:\n        query = query.transpose(1, 2)\n        key = key.transpose(1, 2)\n        value = value.transpose(1, 2)\n        # SageAttention3's Blackwell kernel assumes MHA (Hq == Hkv). For GQA/MQA\n        # (Hq != Hkv), fall back to torch SDPA which supports GQA.\n        if key.shape[1] != query.shape[1]:\n            if query.shape[1] % key.shape[1] != 0:\n                raise ValueError(\n                    \"GQA/MQA requires query heads to be a multiple of KV heads, \"\n                    f\"got q_heads={query.shape[1]} and kv_heads={key.shape[1]}\"\n                )\n            if not type(self)._warned_gqa_fallback_global:\n                logger.warning(\n                    \"SageAttention3 does not support GQA/MQA (Hq != Hkv); falling back to torch SDPA.\"\n                )\n                type(self)._warned_gqa_fallback_global = True\n            output = F.scaled_dot_product_attention(\n                query,\n                key,\n                value,\n                is_causal=self.causal,\n                dropout_p=self.dropout,\n                scale=self.softmax_scale,\n                enable_gqa=True,\n            )\n        else:","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/sage_attn3.py#L53-L89","documentation":"SageAttention3's Blackwell kernel assumes MHA (equal query and KV head counts); for GQA/MQA it falls back to torch SDPA, but that fallback requires query heads to be an integer multiple of KV heads. Otherwise attention grouping is undefined and forward raises ValueError.","triggerScenarios":"Calling forward on the sage_attn3 backend with q.shape[1] % k.shape[1] != 0, e.g. 12 query heads and 8 KV heads.","commonSituations":"Loading a model with unusual head-group sizes not supported by SageAttention3; quantized/remapped attention with mismatched heads; new model configs hitting the GQA fallback path.","solutions":["Fix the model/config head counts so q_heads is a multiple of kv_heads","Use a backend with native GQA support (flash_attn) for this model","If heads are genuinely non-divisible, select a backend supporting arbitrary grouped attention"],"exampleFix":"# before\nimpl = SageAttention3Impl(head_size=128, num_heads=12, num_kv_heads=8, ...)  # 12 % 8 != 0\n# after\nimpl = FlashAttentionImpl(head_size=128, num_heads=12, num_kv_heads=8, ...)","handlingStrategy":"validation","validationCode":"assert query.shape[1] % key.shape[1] == 0, \"q_heads must be a multiple of kv_heads for sage_attn3 fallback\"","typeGuard":"def sage3_compatible(q: torch.Tensor, k: torch.Tensor) -> bool:\n    return q.shape[1] == k.shape[1] or q.shape[1] % k.shape[1] == 0","tryCatchPattern":"try:\n    out = impl.forward(q, k, v, meta)\nexcept ValueError as e:\n    if \"GQA/MQA\" in str(e):\n        out = torch.nn.functional.scaled_dot_product_attention(q, k, v, is_causal=True)","preventionTips":["Check model head-group divisibility before selecting sage_attn3","Prefer flash_attn for exotic GQA ratios"],"tags":["sage-attention","gqa","head-mismatch","validation"],"backgroundTag":"gqa-head-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}