sgl-project/sglang · critical · ValueError

MiniMax H3 attention heads must be divisible by TP size: {ar

Error message

MiniMax H3 attention heads must be divisible by TP size: {arch.num_attention_heads} % {self.tp_size} != 0

What it means

MiniMax H3 partitions attention heads across tensor-parallel ranks; each rank must get an equal integer share of arch.num_attention_heads. If the head count is not divisible by the TP world size, head sharding is impossible and the constructor refuses to build the module.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py:674

_minimax_h3_attention_core_bcg = eager_on_graph(True)(_minimax_h3_attention_core_impl)


class MiniMaxH3Attention(nn.Module):
    def __init__(
        self,
        arch: MiniMaxH3DiTArchConfig,
        quant_config: QuantizationConfig | None,
        *,
        prefix: str,
        bcg_breakpoint: bool = True,
    ) -> None:
        super().__init__()
        self.bcg_breakpoint = bcg_breakpoint
        self.tp_size = get_tp_world_size()
        if arch.num_attention_heads % self.tp_size:
            raise ValueError(
                "MiniMax H3 attention heads must be divisible by TP size: "
                f"{arch.num_attention_heads} % {self.tp_size} != 0"
            )
        self.total_num_heads = arch.num_attention_heads
        self.num_heads = self.total_num_heads // self.tp_size
        self.head_dim = arch.attention_head_dim
        self.inner_dim = self.total_num_heads * self.head_dim
        self.local_inner_dim = self.num_heads * self.head_dim
        self.softmax_scale = self.head_dim**-0.5
        self.prefix = prefix
        self._attention_impl = None
        self._attention_backend_enum: AttentionBackendEnum | None = None
        # The checkpoint stores one fused qkv tensor. Each logical Q/K/V
        # matrix must be sharded independently; a plain ColumnParallelLinear
        # would instead slice across the concatenated tensor and is incorrect
        # for TP > 1.
        self.qkv_proj = MergedColumnParallelLinear(
            arch.hidden_size,

View on GitHub (pinned to 0132848349)

Solutions

1. Use a TP size that divides num_attention_heads (commonly 1, 2, 4, 8)
2. Verify the checkpoint config's num_attention_heads matches the model variant you intend
3. If you control the architecture config, choose a head count with many small factors
4. Fail fast in server args validation with a clear message before weight loading

Example fix

# before
python -m sglang.launch_server --model ... --tp 5
# after
python -m sglang.launch_server --model ... --tp 8

tp = get_tp_world_size()
assert arch.num_attention_heads % tp == 0, f"choose TP in divisors of {arch.num_attention_heads}"

def tp_supported(num_heads: int, tp: int) -> bool:
    return num_heads % tp == 0

When it happens

Trigger: Instantiating the MiniMax H3 attention module with arch.num_attention_heads % get_tp_world_size() != 0, e.g. 48 heads with tp_size=5.

Common situations: Launching the server with --tp / tensor-parallel-size that doesn't divide the model's head count (odd TP sizes like 3/5/7), loading a checkpoint whose config head count differs from expectations, or changing TP degree for a model with a prime-ish head count.

Related errors

• num_heads ({self.num_heads}) must be divisible by tp_size ({
• LTX2Attention requires heads divisible by tp_size, got {self
• LTX2Attention requires inner_dim divisible by tp_size, got {
• out_channels must be divisible by tp_size for TP-sharded out
• adaln out_features mismatch: {out_features} != {expand_ratio

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/872f34466d85e439. Report an issue: GitHub.