sgl-project/sglang · error · ValueError

num_heads ({self.num_heads}) must be divisible by tp_size ({

Error message

num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size}).

What it means

In mova_video_dit's attention module __init__, num_heads must be divisible by the tensor-parallel world size so heads can be sharded evenly across ColumnParallelLinear Q/K/V.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/mova_video_dit.py:112

    USPAttention internally handles the all-to-all communication for distributed attention.
    Input x should already be the local shard [B, S_local, D] when SP is enabled.
    """

    def __init__(
        self,
        dim: int,
        num_heads: int,
        eps: float = 1e-6,
        quant_config: QuantizationConfig | None = None,
    ):
        super().__init__()
        self.dim = dim
        self.num_heads = num_heads
        self.head_dim = dim // num_heads

        self.tp_size = get_tp_world_size()
        if self.num_heads % self.tp_size != 0:
            raise ValueError(
                f"num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size})."
            )
        self.num_heads_per_rank = self.num_heads // self.tp_size

        # TP strategy: shard Q/K/V over heads (column-parallel), then row-parallel output.
        self.q = ColumnParallelLinear(
            dim, dim, bias=True, gather_output=False, quant_config=quant_config
        )
        self.k = ColumnParallelLinear(
            dim, dim, bias=True, gather_output=False, quant_config=quant_config
        )
        self.v = ColumnParallelLinear(
            dim, dim, bias=True, gather_output=False, quant_config=quant_config
        )
        self.o = RowParallelLinear(
            dim, dim, bias=True, input_is_parallel=True, quant_config=quant_config
        )
        self.norm_q = RMSNorm(dim, eps=eps)

View on GitHub (pinned to 0132848349)

Solutions

Choose a TP degree that divides num_heads (commonly a power of two <= num_heads)
Reduce tp_size to a divisor of num_heads
If heads are small (e.g. 24), use TP in {1,2,3,4,6,8→only if divisible} per the head count

Example fix

# before: heads=24, tp=8 -> 24 % 8 != 0
python -m ... --tp 8
# after
python -m ... --tp 6   # or 4/3/2/1

Defensive patterns

Strategy: validation

Validate before calling

tp = get_tp_world_size()
assert num_heads % tp == 0, (num_heads, tp)

Prevention

Choose TP from divisors of num_heads before launch
Validate TP vs head count in a startup sanity check

When it happens

Trigger: Launching with TP degree that does not divide num_heads, e.g. num_heads=24 with tp_size=8 (24%8!=0) or any TP > num_heads.

Common situations: Raising --tp-size beyond head count; model configs with non-power-of-two heads run at high TP; default TP from cluster config mismatching the model.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1f93b2d23e39f92e. Report an issue: GitHub.