{"record":{"id":"ebe1b5c07771b579","repo":"sgl-project/sglang","slug":"minimax-h3-name-value-must-be-divisible-by-tp","errorCode":null,"errorMessage":"MiniMax H3 {name}={value} must be divisible by TP size {tp_size}.","messagePattern":"MiniMax H3 (.+?)=(.+?) must be divisible by TP size (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py","lineNumber":1825,"sourceCode":"            raise ValueError(\"num_attention_heads must be positive.\")\n        if arch.hidden_size <= 0:\n            raise ValueError(\"hidden_size must be positive.\")\n        if arch.attention_head_dim <= 0:\n            raise ValueError(\"attention_head_dim must be positive.\")\n        if arch.ffn_hidden_size <= 0:\n            raise ValueError(\"ffn_hidden_size must be positive.\")\n        for name, value in (\n            (\"num_attention_heads\", arch.num_attention_heads),\n            (\"hidden_size\", arch.hidden_size),\n            (\"ffn_hidden_size\", arch.ffn_hidden_size),\n            (\"time_embed_hidden_size\", arch.time_embed_hidden_size),\n            (\"adaln_out_features\", arch.adaln_out_features),\n            (\"final_adaln_out_features\", arch.final_adaln_out_features),\n            (\"video_patch_output_dim\", arch.latents_dim * math.prod(arch.patch_size)),\n            (\"audio_patch_output_dim\", arch.audio_latents_dim),\n        ):\n            if value % tp_size:\n                raise ValueError(\n                    f\"MiniMax H3 {name}={value} must be divisible by \"\n                    f\"TP size {tp_size}.\"\n                )\n\n    @staticmethod\n    def _validate_sequence_parallel_config(\n        *,\n        arch: MiniMaxH3DiTArchConfig,\n        tp_size: int,\n        ulysses_size: int,\n        ring_size: int,\n    ) -> None:\n        if ulysses_size <= 0:\n            raise ValueError(\"MiniMax H3 Ulysses size must be positive.\")\n        if ring_size <= 0:\n            raise ValueError(\"MiniMax H3 ring size must be positive.\")\n        local_heads = arch.num_attention_heads // tp_size\n        if local_heads % ulysses_size:","sourceCodeStart":1807,"sourceCodeEnd":1843,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py#L1807-L1843","documentation":"Every listed dimension (num_attention_heads, hidden_size, ffn_hidden_size, time_embed_hidden_size, adaln_out_features, final_adaln_out_features, video_patch_output_dim = latents_dim * prod(patch_size), audio_patch_output_dim) must be evenly divisible by the tensor-parallel size, because TP shards each of these linear/attention weights per rank. If any value % tp_size != 0 the model cannot be split, so construction aborts with the offending name and value.","triggerScenarios":"Running with a TP size that does not divide one of the dims, e.g. num_attention_heads=50 with tp_size=4, or a latents_dim/patch_size combination making video_patch_output_dim=1000 with tp_size=3.","commonSituations":"Choosing --tp 3 or 6 on hardware counts (3/6 GPUs) for a model whose widths are powers of two; custom fine-tuned checkpoints with odd head counts; changing patch_size or latents_dim without re-checking TP divisibility.","solutions":["Pick a TP size that divides every listed dim — for stock configs usually a power of two (1/2/4/8)","Re-check video_patch_output_dim = latents_dim * prod(patch_size) specifically if you changed patching","Compute the gcd of all listed dims and choose tp_size among its divisors"],"exampleFix":"# before\n# heads=50, hidden=3072 -> tp=4 fails (50 % 4)\nlaunch(tp_size=4)\n# after\nlaunch(tp_size=2)  # 50 % 2 == 0 and 3072 % 2 == 0","handlingStrategy":"validation","validationCode":"import math\ndims = [arch.num_attention_heads, arch.hidden_size, arch.ffn_hidden_size,\n        arch.time_embed_hidden_size, arch.adaln_out_features,\n        arch.final_adaln_out_features,\n        arch.latents_dim * math.prod(arch.patch_size), arch.audio_latents_dim]\ntp = math.gcd(*dims)\nassert all(d % tp_size == 0 for d in dims), [d for d in dims if d % tp_size]","typeGuard":"def tp_divides(arch, tp: int) -> bool:\n    import math\n    dims = [arch.num_attention_heads, arch.hidden_size, arch.ffn_hidden_size,\n            arch.latents_dim * math.prod(arch.patch_size), arch.audio_latents_dim]\n    return all(d % tp == 0 for d in dims)","tryCatchPattern":null,"preventionTips":["Prefer power-of-two TP sizes for power-of-two models","Re-validate TP divisibility whenever patch_size or latents_dim changes"],"tags":["tensor-parallel","divisibility","config","minimax-h3","launch"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}