{"record":{"id":"dd4243aaaafe259f","repo":"sgl-project/sglang","slug":"num-attention-heads-must-be-divisible-by-attention","errorCode":null,"errorMessage":"num_attention_heads must be divisible by attention TP","messagePattern":"num_attention_heads must be divisible by attention TP","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/interns2_mobius.py","lineNumber":566,"sourceCode":"):\n    \"\"\"Mobius-owned full-attention constructor reusing Qwen3.5 methods.\"\"\"\n\n    def __init__(\n        self,\n        config: InternS2MobiusTextConfig,\n        layer_id: int,\n        quant_config: QuantizationConfig | None = None,\n        prefix: str = \"\",\n        alt_stream: torch.cuda.Stream | None = None,\n    ) -> None:\n        nn.Module.__init__(self)\n        self.config = config\n        self.hidden_size = config.hidden_size\n        self.attn_tp_rank = get_parallel().attn_tp_rank\n        self.attn_tp_size = get_parallel().attn_tp_size\n        self.total_num_heads = config.num_attention_heads\n        if self.total_num_heads % self.attn_tp_size != 0:\n            raise ValueError(\"num_attention_heads must be divisible by attention TP\")\n        self.num_heads = self.total_num_heads // self.attn_tp_size\n        self.total_num_kv_heads = config.num_key_value_heads\n        if self.total_num_kv_heads >= self.attn_tp_size:\n            if self.total_num_kv_heads % self.attn_tp_size != 0:\n                raise ValueError(\n                    \"num_key_value_heads must be divisible by attention TP\"\n                )\n        elif self.attn_tp_size % self.total_num_kv_heads != 0:\n            raise ValueError(\"attention TP must be divisible by num_key_value_heads\")\n        self.num_kv_heads = max(1, self.total_num_kv_heads // self.attn_tp_size)\n        self.head_dim = config.head_dim or (self.hidden_size // self.num_heads)\n        self.q_size = self.num_heads * self.head_dim\n        self.kv_size = self.num_kv_heads * self.head_dim\n        self.scaling = self.head_dim**-0.5\n        self.max_position_embeddings = getattr(config, \"max_position_embeddings\", 8192)\n        self.rope_theta, rope_scaling = get_rope_config(config)\n        self.partial_rotary_factor = getattr(config, \"partial_rotary_factor\", 1.0)\n        self.layer_id = layer_id","sourceCodeStart":548,"sourceCodeEnd":584,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/interns2_mobius.py#L548-L584","documentation":"The attention module validates that total attention heads divide evenly across attention tensor-parallel ranks. If config.num_attention_heads % attn_tp_size != 0, each rank cannot receive an equal shard of heads.","triggerScenarios":"Constructing the decoder layer with attn_tp_size that does not divide config.num_attention_heads — e.g., 40 heads with --tp-size 8 and attention TP decomposition routing to a non-divisor size.","commonSituations":"Launching with an aggressive TP degree for a small model, using --enable-dp-attention or --dp-size which changes attn_tp_size, or mis-set default config in a fine-tuned variant.","solutions":["Lower or change TP size so it divides num_attention_heads (e.g., use tp=4 instead of 8 for 40 heads)","Check effective attn_tp_size via get_parallel() with your dp-attention flags","Confirm the checkpoint's config.json num_attention_heads matches the released model"],"exampleFix":"# before\npython -m sglang.launch_server --model intern-s2-mobius --tp 8   # 40 heads\n\n# after\npython -m sglang.launch_server --model intern-s2-mobius --tp 5   # 40/5=8 heads per rank","handlingStrategy":"validation","validationCode":"assert config.num_attention_heads % get_parallel().attn_tp_size == 0","typeGuard":"def heads_divisible_by_attn_tp(config, attn_tp_size: int) -> bool:\n    return config.num_attention_heads % attn_tp_size == 0","tryCatchPattern":null,"preventionTips":["Compute valid TP divisors of num_attention_heads before launching","Print effective attn_tp_size when using dp-attention"],"tags":["tensor-parallel","attention","launch-config"],"backgroundTag":"parallelism-divisibility-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}