{"record":{"id":"71c699d747f816f5","repo":"sgl-project/sglang","slug":"minimax-h3-tp-local-heads-local-heads-must-be-di","errorCode":null,"errorMessage":"MiniMax H3 TP-local heads {local_heads} must be divisible by Ulysses size {ulysses_size} (total heads={arch.num_attention_heads}, TP={tp_size}).","messagePattern":"MiniMax H3 TP-local heads (.+?) must be divisible by Ulysses size (.+?) \\(total heads=(.+?), TP=(.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py","lineNumber":1844,"sourceCode":"                    f\"MiniMax H3 {name}={value} must be divisible by \"\n                    f\"TP size {tp_size}.\"\n                )\n\n    @staticmethod\n    def _validate_sequence_parallel_config(\n        *,\n        arch: MiniMaxH3DiTArchConfig,\n        tp_size: int,\n        ulysses_size: int,\n        ring_size: int,\n    ) -> None:\n        if ulysses_size <= 0:\n            raise ValueError(\"MiniMax H3 Ulysses size must be positive.\")\n        if ring_size <= 0:\n            raise ValueError(\"MiniMax H3 ring size must be positive.\")\n        local_heads = arch.num_attention_heads // tp_size\n        if local_heads % ulysses_size:\n            raise ValueError(\n                f\"MiniMax H3 TP-local heads {local_heads} must be divisible by \"\n                f\"Ulysses size {ulysses_size} (total heads=\"\n                f\"{arch.num_attention_heads}, TP={tp_size}).\"\n            )\n        # ring never shards heads (only rows), so it has no head-divisibility\n        # constraint; the packed sequence alignment constant must still\n        # divide the *combined* sequence-parallel size, since ring adds an\n        # outer row split on top of Ulysses's inner one (see forward()).\n        sp_size = ulysses_size * ring_size\n        if MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT % sp_size:\n            raise ValueError(\n                \"MiniMax H3 packed sequence alignment \"\n                f\"{MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT} must be divisible by \"\n                f\"the combined sequence-parallel size {sp_size} \"\n                f\"(ulysses={ulysses_size} x ring={ring_size}). Choose degrees \"\n                \"whose product divides both the TP-local attention heads and \"\n                \"the packed sequence alignment.\"\n            )","sourceCodeStart":1826,"sourceCodeEnd":1862,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py#L1826-L1862","documentation":"After TP shards heads (local_heads = num_attention_heads // tp_size), Ulysses further splits those local heads for sequence-parallel attention, so local_heads must be divisible by ulysses_size. If not, head shards would be fractional and the model refuses to construct, reporting total heads and TP for context.","triggerScenarios":"e.g. num_attention_heads=48, tp_size=8 → local_heads=6, then ulysses_size=4 fails (6 % 4 != 0); or heads=50, tp=5, ulysses=3 (10 % 3).","commonSituations":"Enabling aggressive Ulysses (4/8) on top of high TP; changing TP without re-tuning ulysses; using per-rank configs where heads-per-rank is small.","solutions":["Reduce ulysses_size to a divisor of num_attention_heads // tp_size (compute it: ulysses_max = heads // tp; pick a divisor)","Lower tp_size so more local heads remain for Ulysses to split","Keep ulysses_size=1 if you don't need sequence parallelism"],"exampleFix":"# before\n# heads=48, tp=8 -> local=6; ulysses=4 -> 6 % 4 != 0\nulysses_size=4\n# after\nulysses_size=3  # 6 % 3 == 0 (or 6, 2, 1)","handlingStrategy":"validation","validationCode":"local_heads = arch.num_attention_heads // tp_size\nassert local_heads % ulysses_size == 0, f\"{local_heads} % {ulysses_size}\"","typeGuard":"def ulysses_fits(arch, tp: int, u: int) -> bool:\n    return (arch.num_attention_heads // tp) % u == 0","tryCatchPattern":null,"preventionTips":["Compute local heads first, then choose a divisor as ulysses","Prefer lowering TP before raising Ulysses"],"tags":["ulysses","sequence-parallel","divisibility","tensor-parallel"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}