{"record":{"id":"cda2cd9d7db02bce","repo":"sgl-project/sglang","slug":"tp-local-heads-local-heads-not-divisible-by-ulys","errorCode":null,"errorMessage":"TP-local heads {local_heads} not divisible by Ulysses world size {ulysses_ws} (total heads={self.num_attention_heads}, TP={get_tp_world_size()})","messagePattern":"TP-local heads (.+?) not divisible by Ulysses world size (.+?) \\(total heads=(.+?), TP=(.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py","lineNumber":2477,"sourceCode":"        # slice of the packed sequence), Ulysses second (an inner slice\n        # within this rank's ring chunk). Only Ulysses shards heads inside\n        # attention -- ring instead ring-rotates each rank's local KV chunk\n        # and online-softmax merges partial outputs (see\n        # _minimax_h3_attention_core_impl), so it has no head constraint.\n        ulysses_ws, ulysses_rank = get_ulysses_ctx()\n        ring_ws, ring_rank = get_ring_ctx()\n        sp_ws = ulysses_ws * ring_ws\n        local_seq_len = seq_len\n        if sp_ws > 1:\n            if seq_len % sp_ws:\n                raise ValueError(\n                    f\"packed seq_len {seq_len} not divisible by the combined \"\n                    f\"sequence-parallel world size {sp_ws} \"\n                    f\"(ulysses={ulysses_ws} x ring={ring_ws})\"\n                )\n            local_heads = self.num_attention_heads // get_tp_world_size()\n            if local_heads % ulysses_ws:\n                raise ValueError(\n                    f\"TP-local heads {local_heads} not divisible by Ulysses \"\n                    f\"world size {ulysses_ws} (total heads=\"\n                    f\"{self.num_attention_heads}, TP={get_tp_world_size()})\"\n                )\n            local_seq_len = seq_len // sp_ws\n        ring_chunk_len = local_seq_len * ulysses_ws\n        row_start = ring_rank * ring_chunk_len + ulysses_rank * local_seq_len\n        row_stop = row_start + local_seq_len\n\n        # RoPE and latent projections are row-local before Ulysses exchanges\n        # sequence for heads inside attention. Serving normally prepares the\n        # request-static cache once; direct model callers use this fallback.\n        rope_cache = kwargs.get(\"rope_cache\")\n        if rope_cache is None:\n            self.materialize_mps_non_layer_weights(\"rope\")\n            rope_freqs = self.rope(img_position_ids[:, row_start:row_stop]).to(device)\n            rope_cache = (\n                _rope_cos_sin_cache(rope_freqs, dtype=_BF16_DTYPE),","sourceCodeStart":2459,"sourceCodeEnd":2495,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py#L2459-L2495","documentation":"With Ulysses sequence parallelism, the TP-local head count (num_attention_heads // tp_world_size) must be divisible by the Ulysses world size so heads shard evenly inside attention.","triggerScenarios":"Combining tensor parallelism and Ulysses SP such that (total heads / TP) % ulysses_ws != 0, e.g. 48 heads with TP=2 and ulysses_ws=4 gives 24 % 4 == 0 ok, but TP=4 ulysses_ws=8 gives 12 % 8 != 0 → error.","commonSituations":"Raising ulysses degree on small-head models; mixing DP/TP/SP settings where head counts no longer factor; config combos from a different model applied to this one.","solutions":["Pick ulysses_ws that divides num_attention_heads // tp_world_size (e.g. reduce ulysses degree or lower TP)","Increase TP so local heads factor appropriately only if heads/TP stays divisible by ulysses_ws","Use the model's recommended parallel layout (heads usually powers of two)"],"exampleFix":"# before: tp=4, ulysses=8, heads=48 -> local 12 % 8 != 0\n# after: tp=2, ulysses=8, heads=48 -> local 24 % 8 == 0","handlingStrategy":"validation","validationCode":"local_heads = model.num_attention_heads // get_tp_world_size()\nassert local_heads % ulysses_ws == 0, (local_heads, ulysses_ws)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Factor num_attention_heads before choosing TP x ulysses layout","Use recommended parallel configs from model docs"],"tags":["minimax-h3","ulysses","attention-heads","tensor-parallel"],"backgroundTag":"parallelism-shape-divisibility","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}