{"record":{"id":"48c49234408eddea","repo":"sgl-project/sglang","slug":"num-heads-self-num-heads-must-be-divisible-by-48c492","errorCode":null,"errorMessage":"num_heads ({self.num_heads}) must be divisible by ulysses_degree ({ulysses_world_size}).","messagePattern":"num_heads \\((.+?)\\) must be divisible by ulysses_degree \\((.+?)\\)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/dits/lingbot_world.py","lineNumber":207,"sourceCode":"        if c2ws_plucker_emb is None:\n            return hidden_states\n        if c2ws_plucker_emb.shape != hidden_states.shape:\n            raise ValueError(\n                \"c2ws_plucker_emb shape must match hidden_states shape, \"\n                f\"got {tuple(c2ws_plucker_emb.shape)} vs {tuple(hidden_states.shape)}\"\n            )\n        if scale_shift is None:\n            scale_shift = self.compute_scale_shift(c2ws_plucker_emb)\n        cam_scale, cam_shift = scale_shift\n        return (1.0 + cam_scale) * hidden_states + cam_shift\n\n\nclass LingBotWorldCausalSelfAttention(CausalWanSelfAttention):\n    def __init__(self, *args, **kwargs):\n        super().__init__(*args, **kwargs)\n        ulysses_world_size = max(get_ulysses_parallel_world_size(), 1)\n        if self.num_heads % ulysses_world_size != 0:\n            raise ValueError(\n                f\"num_heads ({self.num_heads}) must be divisible by ulysses_degree ({ulysses_world_size}).\"\n            )\n        self.ulysses_num_heads = self.num_heads // ulysses_world_size\n        self.ulysses_attn = LocalAttention(\n            num_heads=self.ulysses_num_heads,\n            head_size=self.head_dim,\n            dropout_rate=0,\n            softmax_scale=None,\n            causal=False,\n            supported_attention_backends=(\n                AttentionBackendEnum.FA,\n                AttentionBackendEnum.AITER,\n                AttentionBackendEnum.TORCH_SDPA,\n            ),\n        )\n\n    def forward(\n        self,","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/lingbot_world.py#L189-L225","documentation":"With Ulysses sequence parallelism, attention heads are split across the SP group, so the head count must be divisible by the Ulysses world size. LingBotWorldCausalSelfAttention checks this at init when it computes ulysses_num_heads = num_heads // ulysses_world_size.","triggerScenarios":"Launching with --ulysses-size (sequence parallel degree) that does not divide the model's attention head count, e.g. num_heads=30 with ulysses_degree=4 (30 % 4 != 0).","commonSituations":"Scaling SP degree to 4/8 on a model with a non-power-of-two head count; changing attention head config without re-checking parallel layout; TP+SP combos reducing effective local heads.","solutions":["Choose a Ulysses degree that divides num_heads (e.g. 1, 2, 3, 5 for 30 heads; up to num_heads itself)","Move parallel capacity to tensor/ring parallelism instead of Ulysses if the head count is awkward","Verify with get_ulysses_parallel_world_size() in a dry-run before serving"],"exampleFix":"# before\npython -m sglang.launch_server --ulysses-size 4 ...   # heads=30 -> crash\n\n# after\npython -m sglang.launch_server --ulysses-size 5 ...   # 30/5=6 local heads","handlingStrategy":"validation","validationCode":"from sglang.srt.distributed import get_ulysses_parallel_world_size\\nif (deg := get_ulysses_parallel_world_size()) > 1:\\n    assert num_heads % deg == 0, f'{num_heads=} % ulysses {deg}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check divisors of head count before choosing Ulysses degree","Document valid parallel layouts per checkpoint","Dry-run model construction in CI with the target launch config"],"tags":["parallelism","ulysses","attention-heads","lingbot","config-validation"],"backgroundTag":"parallel-degree-not-divisible","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}