sgl-project/sglang · critical · ValueError

num_heads ({self.num_heads}) must be divisible by tp_size ({

Error message

num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size}).

What it means

The dual-tower attention shards Q/K/V over attention heads for tensor parallelism, so the number of heads must divide evenly across TP ranks. At init it queries get_tp_world_size() and raises when num_heads % tp_size != 0, because fractional heads per rank cannot be sharded with ColumnParallelLinear.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/bridges/mova_dual_tower.py:207

class ConditionalCrossAttention(nn.Module):
    """
    Cross-modal attention for dual-tower bridge with Tensor Parallel support.

    This module handles attention between video and audio hidden states,
    which have different sequence lengths.
    """

    def __init__(self, dim: int, kv_dim: int, num_heads: int, eps: float = 1e-6):
        super().__init__()
        self.q_dim = dim
        self.kv_dim = kv_dim
        self.num_heads = num_heads
        self.head_dim = self.q_dim // num_heads

        self.tp_size = get_tp_world_size()
        if self.num_heads % self.tp_size != 0:
            raise ValueError(
                f"num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size})."
            )
        self.num_heads_per_rank = self.num_heads // self.tp_size

        # TP strategy: shard Q/K/V over heads (column-parallel), then row-parallel output.
        self.q = ColumnParallelLinear(dim, dim, bias=True, gather_output=False)
        self.k = ColumnParallelLinear(kv_dim, dim, bias=True, gather_output=False)
        self.v = ColumnParallelLinear(kv_dim, dim, bias=True, gather_output=False)
        self.o = RowParallelLinear(dim, dim, bias=True, input_is_parallel=True)
        self.norm_q = RMSNorm(dim, eps=eps)
        self.norm_k = RMSNorm(dim, eps=eps)

        self.attn = USPAttention(
            num_heads=self.num_heads_per_rank,
            head_size=self.head_dim,
            causal=False,
            softmax_scale=None,
            is_cross_attention=True,

View on GitHub (pinned to 0132848349)

Solutions

  1. Pick a tp_size that divides num_heads evenly (e.g. tp_size=1, 2, 4 for 16 heads)
  2. If you control the config, adjust num_heads to a multiple of your TP degree
  3. Check the upstream model's tower config for its head count before choosing --tp
  4. Fall back to tp_size=1 for this bridge if the head count cannot be changed

Example fix

# before (num_heads=30)
python -m sglang... --tp 8   # raises: 30 % 8 != 0
# after
python -m sglang... --tp 2   # 30 % 2 == 0, or use tp 1/3/5/6/10/15
Defensive patterns

Strategy: validation

Validate before calling

tp = get_tp_world_size()
if num_heads % tp != 0:
    raise SystemExit(f"num_heads={num_heads} not divisible by tp={tp}; pick a divisor like "
                     f"{[t for t in range(1, num_heads+1) if num_heads % t == 0]}")

Type guard

def tp_compatible(num_heads: int, tp_size: int) -> bool:
    return num_heads % tp_size == 0

Try / catch

try:
    bridge = MovaDualTowerBridge(...)
except ValueError as e:
    if "divisible by tp_size" in str(e):
        raise SystemExit("restart with a --tp that divides the tower head count") from e
    raise

Prevention

When it happens

Trigger: Initializing the bridge with num_heads not divisible by the TP world size — e.g. num_heads=24 with tp_size=8 is fine, but num_heads=30 with tp_size=8, or any odd head count with tp_size=2 raises immediately in __init__.

Common situations: Launching the server with --tp 8 (or 4/2) against a model config whose tower head count is not a multiple; changing TP degree for throughput without rechecking per-model head counts; porting a single-GPU model config into a multi-GPU deployment.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7abdb19d895c2a73. Report an issue: GitHub.