sgl-project/sglang · critical · ValueError

MiniMax H3 TP-local heads {local_heads} must be divisible by

Error message

MiniMax H3 TP-local heads {local_heads} must be divisible by Ulysses size {ulysses_size} (total heads={arch.num_attention_heads}, TP={tp_size}).

What it means

After TP shards heads (local_heads = num_attention_heads // tp_size), Ulysses further splits those local heads for sequence-parallel attention, so local_heads must be divisible by ulysses_size. If not, head shards would be fractional and the model refuses to construct, reporting total heads and TP for context.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py:1844

                    f"MiniMax H3 {name}={value} must be divisible by "
                    f"TP size {tp_size}."
                )

    @staticmethod
    def _validate_sequence_parallel_config(
        *,
        arch: MiniMaxH3DiTArchConfig,
        tp_size: int,
        ulysses_size: int,
        ring_size: int,
    ) -> None:
        if ulysses_size <= 0:
            raise ValueError("MiniMax H3 Ulysses size must be positive.")
        if ring_size <= 0:
            raise ValueError("MiniMax H3 ring size must be positive.")
        local_heads = arch.num_attention_heads // tp_size
        if local_heads % ulysses_size:
            raise ValueError(
                f"MiniMax H3 TP-local heads {local_heads} must be divisible by "
                f"Ulysses size {ulysses_size} (total heads="
                f"{arch.num_attention_heads}, TP={tp_size})."
            )
        # ring never shards heads (only rows), so it has no head-divisibility
        # constraint; the packed sequence alignment constant must still
        # divide the *combined* sequence-parallel size, since ring adds an
        # outer row split on top of Ulysses's inner one (see forward()).
        sp_size = ulysses_size * ring_size
        if MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT % sp_size:
            raise ValueError(
                "MiniMax H3 packed sequence alignment "
                f"{MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT} must be divisible by "
                f"the combined sequence-parallel size {sp_size} "
                f"(ulysses={ulysses_size} x ring={ring_size}). Choose degrees "
                "whose product divides both the TP-local attention heads and "
                "the packed sequence alignment."
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce ulysses_size to a divisor of num_attention_heads // tp_size (compute it: ulysses_max = heads // tp; pick a divisor)
2. Lower tp_size so more local heads remain for Ulysses to split
3. Keep ulysses_size=1 if you don't need sequence parallelism

Example fix

# before
# heads=48, tp=8 -> local=6; ulysses=4 -> 6 % 4 != 0
ulysses_size=4
# after
ulysses_size=3  # 6 % 3 == 0 (or 6, 2, 1)

local_heads = arch.num_attention_heads // tp_size
assert local_heads % ulysses_size == 0, f"{local_heads} % {ulysses_size}"

def ulysses_fits(arch, tp: int, u: int) -> bool:
    return (arch.num_attention_heads // tp) % u == 0

When it happens

Trigger: e.g. num_attention_heads=48, tp_size=8 → local_heads=6, then ulysses_size=4 fails (6 % 4 != 0); or heads=50, tp=5, ulysses=3 (10 % 3).

Common situations: Enabling aggressive Ulysses (4/8) on top of high TP; changing TP without re-tuning ulysses; using per-rank configs where heads-per-rank is small.

Related errors

• MiniMax H3 packed sequence alignment {MINIMAX_H3_PACKED_SEQU
• packed seq_len {seq_len} not divisible by the combined seque
• world_size must be positive and divide global_heads
• out_channels must be divisible by tp_size for TP-sharded out
• MiniMax H3 {name}={value} must be divisible by TP size {tp_s

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/71c699d747f816f5. Report an issue: GitHub.