sgl-project/sglang · error · ValueError

TP-local heads {local_heads} not divisible by Ulysses world

Error message

TP-local heads {local_heads} not divisible by Ulysses world size {ulysses_ws} (total heads={self.num_attention_heads}, TP={get_tp_world_size()})

What it means

With Ulysses sequence parallelism, the TP-local head count (num_attention_heads // tp_world_size) must be divisible by the Ulysses world size so heads shard evenly inside attention.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py:2477

        # slice of the packed sequence), Ulysses second (an inner slice
        # within this rank's ring chunk). Only Ulysses shards heads inside
        # attention -- ring instead ring-rotates each rank's local KV chunk
        # and online-softmax merges partial outputs (see
        # _minimax_h3_attention_core_impl), so it has no head constraint.
        ulysses_ws, ulysses_rank = get_ulysses_ctx()
        ring_ws, ring_rank = get_ring_ctx()
        sp_ws = ulysses_ws * ring_ws
        local_seq_len = seq_len
        if sp_ws > 1:
            if seq_len % sp_ws:
                raise ValueError(
                    f"packed seq_len {seq_len} not divisible by the combined "
                    f"sequence-parallel world size {sp_ws} "
                    f"(ulysses={ulysses_ws} x ring={ring_ws})"
                )
            local_heads = self.num_attention_heads // get_tp_world_size()
            if local_heads % ulysses_ws:
                raise ValueError(
                    f"TP-local heads {local_heads} not divisible by Ulysses "
                    f"world size {ulysses_ws} (total heads="
                    f"{self.num_attention_heads}, TP={get_tp_world_size()})"
                )
            local_seq_len = seq_len // sp_ws
        ring_chunk_len = local_seq_len * ulysses_ws
        row_start = ring_rank * ring_chunk_len + ulysses_rank * local_seq_len
        row_stop = row_start + local_seq_len

        # RoPE and latent projections are row-local before Ulysses exchanges
        # sequence for heads inside attention. Serving normally prepares the
        # request-static cache once; direct model callers use this fallback.
        rope_cache = kwargs.get("rope_cache")
        if rope_cache is None:
            self.materialize_mps_non_layer_weights("rope")
            rope_freqs = self.rope(img_position_ids[:, row_start:row_stop]).to(device)
            rope_cache = (
                _rope_cos_sin_cache(rope_freqs, dtype=_BF16_DTYPE),

View on GitHub (pinned to 0132848349)

Solutions

1. Pick ulysses_ws that divides num_attention_heads // tp_world_size (e.g. reduce ulysses degree or lower TP)
2. Increase TP so local heads factor appropriately only if heads/TP stays divisible by ulysses_ws
3. Use the model's recommended parallel layout (heads usually powers of two)

Example fix

# before: tp=4, ulysses=8, heads=48 -> local 12 % 8 != 0
# after: tp=2, ulysses=8, heads=48 -> local 24 % 8 == 0

local_heads = model.num_attention_heads // get_tp_world_size()
assert local_heads % ulysses_ws == 0, (local_heads, ulysses_ws)

When it happens

Trigger: Combining tensor parallelism and Ulysses SP such that (total heads / TP) % ulysses_ws != 0, e.g. 48 heads with TP=2 and ulysses_ws=4 gives 24 % 4 == 0 ok, but TP=4 ulysses_ws=8 gives 12 % 8 != 0 → error.

Common situations: Raising ulysses degree on small-head models; mixing DP/TP/SP settings where head counts no longer factor; config combos from a different model applied to this one.

Related errors

• Cosmos3CausalAttention requires num_attention_heads divisibl
• num_heads ({self.num_heads}) must be divisible by ulysses_de
• LTX2Attention requires heads divisible by tp_size, got {self
• MiniMax H3 attention heads must be divisible by TP size: {ar
• TP size must be positive.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cda2cd9d7db02bce. Report an issue: GitHub.