{"record":{"id":"872f34466d85e439","repo":"sgl-project/sglang","slug":"minimax-h3-attention-heads-must-be-divisible-by-tp","errorCode":null,"errorMessage":"MiniMax H3 attention heads must be divisible by TP size: {arch.num_attention_heads} % {self.tp_size} != 0","messagePattern":"MiniMax H3 attention heads must be divisible by TP size: (.+?) % (.+?) != 0","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py","lineNumber":674,"sourceCode":"\n\n_minimax_h3_attention_core_bcg = eager_on_graph(True)(_minimax_h3_attention_core_impl)\n\n\nclass MiniMaxH3Attention(nn.Module):\n    def __init__(\n        self,\n        arch: MiniMaxH3DiTArchConfig,\n        quant_config: QuantizationConfig | None,\n        *,\n        prefix: str,\n        bcg_breakpoint: bool = True,\n    ) -> None:\n        super().__init__()\n        self.bcg_breakpoint = bcg_breakpoint\n        self.tp_size = get_tp_world_size()\n        if arch.num_attention_heads % self.tp_size:\n            raise ValueError(\n                \"MiniMax H3 attention heads must be divisible by TP size: \"\n                f\"{arch.num_attention_heads} % {self.tp_size} != 0\"\n            )\n        self.total_num_heads = arch.num_attention_heads\n        self.num_heads = self.total_num_heads // self.tp_size\n        self.head_dim = arch.attention_head_dim\n        self.inner_dim = self.total_num_heads * self.head_dim\n        self.local_inner_dim = self.num_heads * self.head_dim\n        self.softmax_scale = self.head_dim**-0.5\n        self.prefix = prefix\n        self._attention_impl = None\n        self._attention_backend_enum: AttentionBackendEnum | None = None\n        # The checkpoint stores one fused qkv tensor. Each logical Q/K/V\n        # matrix must be sharded independently; a plain ColumnParallelLinear\n        # would instead slice across the concatenated tensor and is incorrect\n        # for TP > 1.\n        self.qkv_proj = MergedColumnParallelLinear(\n            arch.hidden_size,","sourceCodeStart":656,"sourceCodeEnd":692,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py#L656-L692","documentation":"MiniMax H3 partitions attention heads across tensor-parallel ranks; each rank must get an equal integer share of arch.num_attention_heads. If the head count is not divisible by the TP world size, head sharding is impossible and the constructor refuses to build the module.","triggerScenarios":"Instantiating the MiniMax H3 attention module with arch.num_attention_heads % get_tp_world_size() != 0, e.g. 48 heads with tp_size=5.","commonSituations":"Launching the server with --tp / tensor-parallel-size that doesn't divide the model's head count (odd TP sizes like 3/5/7), loading a checkpoint whose config head count differs from expectations, or changing TP degree for a model with a prime-ish head count.","solutions":["Use a TP size that divides num_attention_heads (commonly 1, 2, 4, 8)","Verify the checkpoint config's num_attention_heads matches the model variant you intend","If you control the architecture config, choose a head count with many small factors","Fail fast in server args validation with a clear message before weight loading"],"exampleFix":"# before\npython -m sglang.launch_server --model ... --tp 5\n# after\npython -m sglang.launch_server --model ... --tp 8","handlingStrategy":"validation","validationCode":"tp = get_tp_world_size()\nassert arch.num_attention_heads % tp == 0, f\"choose TP in divisors of {arch.num_attention_heads}\"","typeGuard":"def tp_supported(num_heads: int, tp: int) -> bool:\n    return num_heads % tp == 0","tryCatchPattern":null,"preventionTips":["Validate TP against head count in server args before launching","Prefer power-of-two TP sizes","Keep a per-model table of valid TP degrees"],"tags":["minimax-h3","tensor-parallel","config-validation","startup"],"backgroundTag":"tensor-parallel-head-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}