{"record":{"id":"a756bd33d44c831b","repo":"sgl-project/sglang","slug":"num-key-value-heads-must-be-divisible-by-attention","errorCode":null,"errorMessage":"num_key_value_heads must be divisible by attention TP","messagePattern":"num_key_value_heads must be divisible by attention TP","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/interns2_mobius.py","lineNumber":571,"sourceCode":"        config: InternS2MobiusTextConfig,\n        layer_id: int,\n        quant_config: QuantizationConfig | None = None,\n        prefix: str = \"\",\n        alt_stream: torch.cuda.Stream | None = None,\n    ) -> None:\n        nn.Module.__init__(self)\n        self.config = config\n        self.hidden_size = config.hidden_size\n        self.attn_tp_rank = get_parallel().attn_tp_rank\n        self.attn_tp_size = get_parallel().attn_tp_size\n        self.total_num_heads = config.num_attention_heads\n        if self.total_num_heads % self.attn_tp_size != 0:\n            raise ValueError(\"num_attention_heads must be divisible by attention TP\")\n        self.num_heads = self.total_num_heads // self.attn_tp_size\n        self.total_num_kv_heads = config.num_key_value_heads\n        if self.total_num_kv_heads >= self.attn_tp_size:\n            if self.total_num_kv_heads % self.attn_tp_size != 0:\n                raise ValueError(\n                    \"num_key_value_heads must be divisible by attention TP\"\n                )\n        elif self.attn_tp_size % self.total_num_kv_heads != 0:\n            raise ValueError(\"attention TP must be divisible by num_key_value_heads\")\n        self.num_kv_heads = max(1, self.total_num_kv_heads // self.attn_tp_size)\n        self.head_dim = config.head_dim or (self.hidden_size // self.num_heads)\n        self.q_size = self.num_heads * self.head_dim\n        self.kv_size = self.num_kv_heads * self.head_dim\n        self.scaling = self.head_dim**-0.5\n        self.max_position_embeddings = getattr(config, \"max_position_embeddings\", 8192)\n        self.rope_theta, rope_scaling = get_rope_config(config)\n        self.partial_rotary_factor = getattr(config, \"partial_rotary_factor\", 1.0)\n        self.layer_id = layer_id\n        if rope_scaling and not (\"rope_type\" in rope_scaling or \"type\" in rope_scaling):\n            rope_scaling = None\n        self.attn_output_gate = getattr(config, \"attn_output_gate\", True)\n        self.rotary_emb = get_rope(\n            head_size=self.head_dim,","sourceCodeStart":553,"sourceCodeEnd":589,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/interns2_mobius.py#L553-L589","documentation":"When total_num_kv_heads >= attn_tp_size, KV heads must shard evenly: num_key_value_heads % attn_tp_size == 0. Otherwise replication/sharding of GQA KV heads is undefined.","triggerScenarios":"Building the attention with attn_tp_size exceeding but not dividing num_key_value_heads, e.g., 6 KV heads with attn_tp_size 8 (6 < 8 falls to the elif, but 12 KV heads with tp=8 hits this branch: 12%8!=0).","commonSituations":"GQA models with few KV heads launched under high TP, or DP-attention setups that change the effective attention TP group size.","solutions":["Choose TP size that divides num_key_value_heads (e.g., tp=6 or 4 for 12 KV heads)","Enable or adjust --dp-size / dp-attention so attn_tp_size divides KV heads","Fall back to tp=1 for odd head counts"],"exampleFix":"# before\n--tp 8   # 12 kv heads, 12 % 8 != 0\n\n# after\n--tp 4    # 12 % 4 == 0","handlingStrategy":"validation","validationCode":"kv, tp = config.num_key_value_heads, get_parallel().attn_tp_size\nassert kv < tp or kv % tp == 0","typeGuard":"def kv_heads_shard_ok(kv: int, tp: int) -> bool:\n    return kv % tp == 0 if kv >= tp else tp % kv == 0","tryCatchPattern":null,"preventionTips":["Check both head divisibility rules before choosing TP size","Prefer TP sizes that divide all of: heads, kv heads"],"tags":["gqa","kv-heads","tensor-parallel"],"backgroundTag":"parallelism-divisibility-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}