sgl-project/sglang · critical · ValueError

num_heads ({self.num_heads}) must be divisible by ulysses_de

Error message

num_heads ({self.num_heads}) must be divisible by ulysses_degree ({ulysses_world_size}).

What it means

With Ulysses sequence parallelism, attention heads are split across the SP group, so the head count must be divisible by the Ulysses world size. LingBotWorldCausalSelfAttention checks this at init when it computes ulysses_num_heads = num_heads // ulysses_world_size.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/lingbot_world.py:207

        if c2ws_plucker_emb is None:
            return hidden_states
        if c2ws_plucker_emb.shape != hidden_states.shape:
            raise ValueError(
                "c2ws_plucker_emb shape must match hidden_states shape, "
                f"got {tuple(c2ws_plucker_emb.shape)} vs {tuple(hidden_states.shape)}"
            )
        if scale_shift is None:
            scale_shift = self.compute_scale_shift(c2ws_plucker_emb)
        cam_scale, cam_shift = scale_shift
        return (1.0 + cam_scale) * hidden_states + cam_shift


class LingBotWorldCausalSelfAttention(CausalWanSelfAttention):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        ulysses_world_size = max(get_ulysses_parallel_world_size(), 1)
        if self.num_heads % ulysses_world_size != 0:
            raise ValueError(
                f"num_heads ({self.num_heads}) must be divisible by ulysses_degree ({ulysses_world_size})."
            )
        self.ulysses_num_heads = self.num_heads // ulysses_world_size
        self.ulysses_attn = LocalAttention(
            num_heads=self.ulysses_num_heads,
            head_size=self.head_dim,
            dropout_rate=0,
            softmax_scale=None,
            causal=False,
            supported_attention_backends=(
                AttentionBackendEnum.FA,
                AttentionBackendEnum.AITER,
                AttentionBackendEnum.TORCH_SDPA,
            ),
        )

    def forward(
        self,

View on GitHub (pinned to 0132848349)

Solutions

1. Choose a Ulysses degree that divides num_heads (e.g. 1, 2, 3, 5 for 30 heads; up to num_heads itself)
2. Move parallel capacity to tensor/ring parallelism instead of Ulysses if the head count is awkward
3. Verify with get_ulysses_parallel_world_size() in a dry-run before serving

Example fix

# before
python -m sglang.launch_server --ulysses-size 4 ...   # heads=30 -> crash

# after
python -m sglang.launch_server --ulysses-size 5 ...   # 30/5=6 local heads

from sglang.srt.distributed import get_ulysses_parallel_world_size\nif (deg := get_ulysses_parallel_world_size()) > 1:\n    assert num_heads % deg == 0, f'{num_heads=} % ulysses {deg}'

When it happens

Trigger: Launching with --ulysses-size (sequence parallel degree) that does not divide the model's attention head count, e.g. num_heads=30 with ulysses_degree=4 (30 % 4 != 0).

Common situations: Scaling SP degree to 4/8 on a model with a non-power-of-two head count; changing attention head config without re-checking parallel layout; TP+SP combos reducing effective local heads.

Related errors

• LTX2Attention requires heads divisible by tp_size, got {self
• Hidden size {self.hidden_size} must be divisible by num_atte
• Unsupported qk_norm: {qk_norm}
• LingBotWorld requires cross_attn_norm=True
• LTX2Attention requires inner_dim divisible by tp_size, got {

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/48c49234408eddea. Report an issue: GitHub.