{"record":{"id":"7abdb19d895c2a73","repo":"sgl-project/sglang","slug":"num-heads-self-num-heads-must-be-divisible-by","errorCode":null,"errorMessage":"num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size}).","messagePattern":"num_heads \\((.+?)\\) must be divisible by tp_size \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/bridges/mova_dual_tower.py","lineNumber":207,"sourceCode":"\nclass ConditionalCrossAttention(nn.Module):\n    \"\"\"\n    Cross-modal attention for dual-tower bridge with Tensor Parallel support.\n\n    This module handles attention between video and audio hidden states,\n    which have different sequence lengths.\n    \"\"\"\n\n    def __init__(self, dim: int, kv_dim: int, num_heads: int, eps: float = 1e-6):\n        super().__init__()\n        self.q_dim = dim\n        self.kv_dim = kv_dim\n        self.num_heads = num_heads\n        self.head_dim = self.q_dim // num_heads\n\n        self.tp_size = get_tp_world_size()\n        if self.num_heads % self.tp_size != 0:\n            raise ValueError(\n                f\"num_heads ({self.num_heads}) must be divisible by tp_size ({self.tp_size}).\"\n            )\n        self.num_heads_per_rank = self.num_heads // self.tp_size\n\n        # TP strategy: shard Q/K/V over heads (column-parallel), then row-parallel output.\n        self.q = ColumnParallelLinear(dim, dim, bias=True, gather_output=False)\n        self.k = ColumnParallelLinear(kv_dim, dim, bias=True, gather_output=False)\n        self.v = ColumnParallelLinear(kv_dim, dim, bias=True, gather_output=False)\n        self.o = RowParallelLinear(dim, dim, bias=True, input_is_parallel=True)\n        self.norm_q = RMSNorm(dim, eps=eps)\n        self.norm_k = RMSNorm(dim, eps=eps)\n\n        self.attn = USPAttention(\n            num_heads=self.num_heads_per_rank,\n            head_size=self.head_dim,\n            causal=False,\n            softmax_scale=None,\n            is_cross_attention=True,","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/bridges/mova_dual_tower.py#L189-L225","documentation":"The dual-tower attention shards Q/K/V over attention heads for tensor parallelism, so the number of heads must divide evenly across TP ranks. At init it queries get_tp_world_size() and raises when num_heads % tp_size != 0, because fractional heads per rank cannot be sharded with ColumnParallelLinear.","triggerScenarios":"Initializing the bridge with num_heads not divisible by the TP world size — e.g. num_heads=24 with tp_size=8 is fine, but num_heads=30 with tp_size=8, or any odd head count with tp_size=2 raises immediately in __init__.","commonSituations":"Launching the server with --tp 8 (or 4/2) against a model config whose tower head count is not a multiple; changing TP degree for throughput without rechecking per-model head counts; porting a single-GPU model config into a multi-GPU deployment.","solutions":["Pick a tp_size that divides num_heads evenly (e.g. tp_size=1, 2, 4 for 16 heads)","If you control the config, adjust num_heads to a multiple of your TP degree","Check the upstream model's tower config for its head count before choosing --tp","Fall back to tp_size=1 for this bridge if the head count cannot be changed"],"exampleFix":"# before (num_heads=30)\npython -m sglang... --tp 8   # raises: 30 % 8 != 0\n# after\npython -m sglang... --tp 2   # 30 % 2 == 0, or use tp 1/3/5/6/10/15","handlingStrategy":"validation","validationCode":"tp = get_tp_world_size()\nif num_heads % tp != 0:\n    raise SystemExit(f\"num_heads={num_heads} not divisible by tp={tp}; pick a divisor like \"\n                     f\"{[t for t in range(1, num_heads+1) if num_heads % t == 0]}\")","typeGuard":"def tp_compatible(num_heads: int, tp_size: int) -> bool:\n    return num_heads % tp_size == 0","tryCatchPattern":"try:\n    bridge = MovaDualTowerBridge(...)\nexcept ValueError as e:\n    if \"divisible by tp_size\" in str(e):\n        raise SystemExit(\"restart with a --tp that divides the tower head count\") from e\n    raise","preventionTips":["Check model head counts against planned TP degree before launching multi-GPU jobs","Document valid --tp values per model in deployment runbooks"],"tags":["tensor-parallel","model-config","startup","tp-sharding"],"backgroundTag":"heads-not-divisible-by-tp-size","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}