{"record":{"id":"1f93b2d23e39f92e","repo":"sgl-project/sglang","slug":"num-heads-self-num-heads-must-be-divisible-by-1f93b2","errorCode":null,"errorMessage":"num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size}).","messagePattern":"num_heads \\((.+?)\\) must be divisible by tp_size \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/mova_video_dit.py","lineNumber":112,"sourceCode":"    USPAttention internally handles the all-to-all communication for distributed attention.\n    Input x should already be the local shard [B, S_local, D] when SP is enabled.\n    \"\"\"\n\n    def __init__(\n        self,\n        dim: int,\n        num_heads: int,\n        eps: float = 1e-6,\n        quant_config: QuantizationConfig | None = None,\n    ):\n        super().__init__()\n        self.dim = dim\n        self.num_heads = num_heads\n        self.head_dim = dim // num_heads\n\n        self.tp_size = get_tp_world_size()\n        if self.num_heads % self.tp_size != 0:\n            raise ValueError(\n                f\"num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size}).\"\n            )\n        self.num_heads_per_rank = self.num_heads // self.tp_size\n\n        # TP strategy: shard Q/K/V over heads (column-parallel), then row-parallel output.\n        self.q = ColumnParallelLinear(\n            dim, dim, bias=True, gather_output=False, quant_config=quant_config\n        )\n        self.k = ColumnParallelLinear(\n            dim, dim, bias=True, gather_output=False, quant_config=quant_config\n        )\n        self.v = ColumnParallelLinear(\n            dim, dim, bias=True, gather_output=False, quant_config=quant_config\n        )\n        self.o = RowParallelLinear(\n            dim, dim, bias=True, input_is_parallel=True, quant_config=quant_config\n        )\n        self.norm_q = RMSNorm(dim, eps=eps)","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/mova_video_dit.py#L94-L130","documentation":"In mova_video_dit's attention module __init__, num_heads must be divisible by the tensor-parallel world size so heads can be sharded evenly across ColumnParallelLinear Q/K/V.","triggerScenarios":"Launching with TP degree that does not divide num_heads, e.g. num_heads=24 with tp_size=8 (24%8!=0) or any TP > num_heads.","commonSituations":"Raising --tp-size beyond head count; model configs with non-power-of-two heads run at high TP; default TP from cluster config mismatching the model.","solutions":["Choose a TP degree that divides num_heads (commonly a power of two <= num_heads)","Reduce tp_size to a divisor of num_heads","If heads are small (e.g. 24), use TP in {1,2,3,4,6,8→only if divisible} per the head count"],"exampleFix":"# before: heads=24, tp=8 -> 24 % 8 != 0\npython -m ... --tp 8\n# after\npython -m ... --tp 6   # or 4/3/2/1","handlingStrategy":"validation","validationCode":"tp = get_tp_world_size()\nassert num_heads % tp == 0, (num_heads, tp)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Choose TP from divisors of num_heads before launch","Validate TP vs head count in a startup sanity check"],"tags":["mova-video-dit","tensor-parallel","attention-heads","divisibility"],"backgroundTag":"parallelism-shape-divisibility","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}