{"record":{"id":"7c4923d1e22e8dfe","repo":"sgl-project/sglang","slug":"attention-tp-must-be-divisible-by-num-key-value-he","errorCode":null,"errorMessage":"attention TP must be divisible by num_key_value_heads","messagePattern":"attention TP must be divisible by num_key_value_heads","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/interns2_mobius.py","lineNumber":575,"sourceCode":"        alt_stream: torch.cuda.Stream | None = None,\n    ) -> None:\n        nn.Module.__init__(self)\n        self.config = config\n        self.hidden_size = config.hidden_size\n        self.attn_tp_rank = get_parallel().attn_tp_rank\n        self.attn_tp_size = get_parallel().attn_tp_size\n        self.total_num_heads = config.num_attention_heads\n        if self.total_num_heads % self.attn_tp_size != 0:\n            raise ValueError(\"num_attention_heads must be divisible by attention TP\")\n        self.num_heads = self.total_num_heads // self.attn_tp_size\n        self.total_num_kv_heads = config.num_key_value_heads\n        if self.total_num_kv_heads >= self.attn_tp_size:\n            if self.total_num_kv_heads % self.attn_tp_size != 0:\n                raise ValueError(\n                    \"num_key_value_heads must be divisible by attention TP\"\n                )\n        elif self.attn_tp_size % self.total_num_kv_heads != 0:\n            raise ValueError(\"attention TP must be divisible by num_key_value_heads\")\n        self.num_kv_heads = max(1, self.total_num_kv_heads // self.attn_tp_size)\n        self.head_dim = config.head_dim or (self.hidden_size // self.num_heads)\n        self.q_size = self.num_heads * self.head_dim\n        self.kv_size = self.num_kv_heads * self.head_dim\n        self.scaling = self.head_dim**-0.5\n        self.max_position_embeddings = getattr(config, \"max_position_embeddings\", 8192)\n        self.rope_theta, rope_scaling = get_rope_config(config)\n        self.partial_rotary_factor = getattr(config, \"partial_rotary_factor\", 1.0)\n        self.layer_id = layer_id\n        if rope_scaling and not (\"rope_type\" in rope_scaling or \"type\" in rope_scaling):\n            rope_scaling = None\n        self.attn_output_gate = getattr(config, \"attn_output_gate\", True)\n        self.rotary_emb = get_rope(\n            head_size=self.head_dim,\n            rotary_dim=self.head_dim,\n            max_position=self.max_position_embeddings,\n            rope_scaling=rope_scaling,\n            base=self.rope_theta,","sourceCodeStart":557,"sourceCodeEnd":593,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/interns2_mobius.py#L557-L593","documentation":"When attn_tp_size > total_num_kv_heads, each KV head must be replicated across an integer number of ranks: attn_tp_size % num_key_value_heads == 0. Otherwise some ranks would own fractional KV heads.","triggerScenarios":"e.g., 3 KV heads with attn_tp_size 8 (8 % 3 != 0) — replication factor would be non-integral.","commonSituations":"Deep GQA models (very few KV heads) under large TP degrees; enabling dp-attention which shrinks attn_tp_size can inadvertently avoid or trigger this.","solutions":["Pick TP size that is a multiple of num_key_value_heads (tp=3 or 6 for 3 KV heads)","Use --dp-size to scale down attn_tp_size to a divisor/multiple relationship","Serve with lower TP such that attn_tp_size <= num_key_value_heads and divides it"],"exampleFix":"# before\n--tp 8   # 3 kv heads: 8 % 3 != 0\n\n# after\n--tp 3    # exact replication factor 1","handlingStrategy":"validation","validationCode":"kv, tp = config.num_key_value_heads, get_parallel().attn_tp_size\nassert kv >= tp or tp % kv == 0","typeGuard":"def kv_replication_ok(kv: int, tp: int) -> bool:\n    return tp % kv == 0","tryCatchPattern":null,"preventionTips":["For few-KV-head GQA models, choose TP that is a multiple of kv heads","Document valid TP degrees per model in launch scripts"],"tags":["gqa","kv-heads","tensor-parallel"],"backgroundTag":"parallelism-divisibility-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}