sgl-project/sglang · critical · ValueError

MiniMax H3 {name}={value} must be divisible by TP size {tp_s

Error message

MiniMax H3 {name}={value} must be divisible by TP size {tp_size}.

What it means

Every listed dimension (num_attention_heads, hidden_size, ffn_hidden_size, time_embed_hidden_size, adaln_out_features, final_adaln_out_features, video_patch_output_dim = latents_dim * prod(patch_size), audio_patch_output_dim) must be evenly divisible by the tensor-parallel size, because TP shards each of these linear/attention weights per rank. If any value % tp_size != 0 the model cannot be split, so construction aborts with the offending name and value.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py:1825

            raise ValueError("num_attention_heads must be positive.")
        if arch.hidden_size <= 0:
            raise ValueError("hidden_size must be positive.")
        if arch.attention_head_dim <= 0:
            raise ValueError("attention_head_dim must be positive.")
        if arch.ffn_hidden_size <= 0:
            raise ValueError("ffn_hidden_size must be positive.")
        for name, value in (
            ("num_attention_heads", arch.num_attention_heads),
            ("hidden_size", arch.hidden_size),
            ("ffn_hidden_size", arch.ffn_hidden_size),
            ("time_embed_hidden_size", arch.time_embed_hidden_size),
            ("adaln_out_features", arch.adaln_out_features),
            ("final_adaln_out_features", arch.final_adaln_out_features),
            ("video_patch_output_dim", arch.latents_dim * math.prod(arch.patch_size)),
            ("audio_patch_output_dim", arch.audio_latents_dim),
        ):
            if value % tp_size:
                raise ValueError(
                    f"MiniMax H3 {name}={value} must be divisible by "
                    f"TP size {tp_size}."
                )

    @staticmethod
    def _validate_sequence_parallel_config(
        *,
        arch: MiniMaxH3DiTArchConfig,
        tp_size: int,
        ulysses_size: int,
        ring_size: int,
    ) -> None:
        if ulysses_size <= 0:
            raise ValueError("MiniMax H3 Ulysses size must be positive.")
        if ring_size <= 0:
            raise ValueError("MiniMax H3 ring size must be positive.")
        local_heads = arch.num_attention_heads // tp_size
        if local_heads % ulysses_size:

View on GitHub (pinned to 0132848349)

Solutions

Pick a TP size that divides every listed dim — for stock configs usually a power of two (1/2/4/8)
Re-check video_patch_output_dim = latents_dim * prod(patch_size) specifically if you changed patching
Compute the gcd of all listed dims and choose tp_size among its divisors

Example fix

# before
# heads=50, hidden=3072 -> tp=4 fails (50 % 4)
launch(tp_size=4)
# after
launch(tp_size=2)  # 50 % 2 == 0 and 3072 % 2 == 0

Defensive patterns

Strategy: validation

Validate before calling

import math
dims = [arch.num_attention_heads, arch.hidden_size, arch.ffn_hidden_size,
        arch.time_embed_hidden_size, arch.adaln_out_features,
        arch.final_adaln_out_features,
        arch.latents_dim * math.prod(arch.patch_size), arch.audio_latents_dim]
tp = math.gcd(*dims)
assert all(d % tp_size == 0 for d in dims), [d for d in dims if d % tp_size]

Type guard

def tp_divides(arch, tp: int) -> bool:
    import math
    dims = [arch.num_attention_heads, arch.hidden_size, arch.ffn_hidden_size,
            arch.latents_dim * math.prod(arch.patch_size), arch.audio_latents_dim]
    return all(d % tp == 0 for d in dims)

Prevention

Prefer power-of-two TP sizes for power-of-two models
Re-validate TP divisibility whenever patch_size or latents_dim changes

When it happens

Trigger: Running with a TP size that does not divide one of the dims, e.g. num_attention_heads=50 with tp_size=4, or a latents_dim/patch_size combination making video_patch_output_dim=1000 with tp_size=3.

Common situations: Choosing --tp 3 or 6 on hardware counts (3/6 GPUs) for a model whose widths are powers of two; custom fine-tuned checkpoints with odd head counts; changing patch_size or latents_dim without re-checking TP divisibility.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ebe1b5c07771b579. Report an issue: GitHub.