{"record":{"id":"edaebb3ee21d1f6b","repo":"sgl-project/sglang","slug":"aiter-backend-requires-num-heads-num-heads-to","errorCode":null,"errorMessage":"AITer backend requires num_heads ({num_heads}) to be a multiple of num_kv_heads ({num_kv_heads}).","messagePattern":"AITer backend requires num_heads \\((.+?)\\) to be a multiple of num_kv_heads \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/aiter.py","lineNumber":136,"sourceCode":"    \"\"\"\n\n    def __init__(\n        self,\n        num_heads: int,\n        head_size: int,\n        softmax_scale: float,\n        causal: bool = False,\n        num_kv_heads: int | None = None,\n        prefix: str = \"\",\n        dropout_p: float = 0.0,\n        **extra_impl_args,\n    ) -> None:\n        # aiter's mha entry points take GQA/MQA K/V directly (they broadcast\n        # each KV head across its group of query heads), so the only\n        # requirement is an even split. The FP8 ASM path is MHA-only and\n        # already routes grouped shapes back to BF16 below.\n        if num_kv_heads is not None and num_heads % num_kv_heads != 0:\n            raise ValueError(\n                f\"AITer backend requires num_heads ({num_heads}) to be a \"\n                f\"multiple of num_kv_heads ({num_kv_heads}).\"\n            )\n        self.causal = causal\n        self.dropout_p = dropout_p\n        self.softmax_scale = softmax_scale\n\n    @torch.compiler.disable\n    def forward(\n        self,\n        query: torch.Tensor,\n        key: torch.Tensor,\n        value: torch.Tensor,\n        attn_metadata: AttentionMetadata | None = None,\n    ) -> torch.Tensor:\n        \"\"\"\n        Performs attention using one of:\n          - _fmha_fp8_prefill_attention (FP8, SGLANG_DIFFUSION_AITER_FP8_ATTN=1 when eligible)","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/aiter.py#L118-L154","documentation":"AITerImpl.__init__ validates that num_heads divides evenly by num_kv_heads (GQA constraint). aiter's MHA kernels broadcast each KV head across its query-head group, so an uneven split cannot be mapped. num_kv_heads=None skips the check.","triggerScenarios":"Constructing AITerImpl with num_kv_heads not None and num_heads % num_kv_heads != 0, e.g. num_heads=24, num_kv_heads=8 (24%8==0 passes) vs num_heads=28, num_kv_heads=6 (fails). Typically the values come from the model's attention config (num_attention_heads / num_key_value_heads).","commonSituations":"Selecting the aiter attention backend on AMD GPUs for a model whose GQA head grouping is fractional (query heads not a multiple of KV heads), or a config typo in num_key_value_heads.","solutions":["Use a model/config where num_attention_heads is a multiple of num_key_value_heads (standard GQA shapes)","Fix the num_heads/num_kv_heads values being passed if they were misread from the model config","Switch to a backend that supports uneven GQA grouping (e.g. flash attention) for this model"],"exampleFix":"// before: model config has 28 query heads, 6 kv heads -> 28 % 6 != 0\nAITerImpl(num_heads=28, head_size=128, softmax_scale=s, num_kv_heads=6)\n// after: use a valid GQA shape or another backend\nAITerImpl(num_heads=32, head_size=128, softmax_scale=s, num_kv_heads=8)  # 32 % 8 == 0","handlingStrategy":"validation","validationCode":"if num_kv_heads is not None:\n    assert num_heads % num_kv_heads == 0, (\n        f\"invalid GQA shape: {num_heads} heads / {num_kv_heads} kv heads\"\n    )","typeGuard":"def is_valid_gqa(num_heads: int, num_kv_heads: int | None) -> bool:\n    return num_kv_heads is None or num_heads % num_kv_heads == 0","tryCatchPattern":null,"preventionTips":["Validate model GQA shapes once at config load time","Cross-check num_attention_heads and num_key_value_heads from the model's config.json"],"tags":["aiter","gqa","attention-backend","rocm","shape-validation"],"backgroundTag":"head-dimension-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}