{"record":{"id":"cfcfdf87c013b63d","repo":"sgl-project/sglang","slug":"cosmos3causalattention-requires-num-attention-head","errorCode":null,"errorMessage":"Cosmos3CausalAttention requires num_attention_heads divisible by tp_size, got {num_attention_heads=} {self.tp_size=}.","messagePattern":"Cosmos3CausalAttention requires num_attention_heads divisible by tp_size, got (.+?) (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/cosmos3video.py","lineNumber":585,"sourceCode":"        hidden_size: int,\n        num_attention_heads: int,\n        num_key_value_heads: int,\n        head_dim: int,\n        qk_norm: bool = True,\n        use_k_norm_und_for_gen: bool = False,\n        rms_norm_eps: float = 1e-6,\n        prefix: str = \"\",\n        quant_config: QuantizationConfig | None = None,\n    ):\n        super().__init__()\n        self.hidden_size = hidden_size\n        self.num_attention_heads = num_attention_heads\n        self.num_key_value_heads = num_key_value_heads\n        self.head_dim = head_dim\n        self.qk_norm = qk_norm\n        self.tp_size = get_tp_world_size()\n        if num_attention_heads % self.tp_size != 0:\n            raise ValueError(\n                \"Cosmos3CausalAttention requires num_attention_heads divisible \"\n                f\"by tp_size, got {num_attention_heads=} {self.tp_size=}.\"\n            )\n        if num_key_value_heads % self.tp_size != 0:\n            raise ValueError(\n                \"Cosmos3CausalAttention requires num_key_value_heads divisible \"\n                f\"by tp_size, got {num_key_value_heads=} {self.tp_size=}.\"\n            )\n        self.local_num_attention_heads = num_attention_heads // self.tp_size\n        self.local_num_key_value_heads = num_key_value_heads // self.tp_size\n\n        self.q_size = num_attention_heads * head_dim\n        self.kv_size = num_key_value_heads * head_dim\n        self.to_qkv = MergedColumnParallelLinear(\n            hidden_size,\n            [self.q_size, self.kv_size, self.kv_size],\n            bias=False,\n            gather_output=False,","sourceCodeStart":567,"sourceCodeEnd":603,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/cosmos3video.py#L567-L603","documentation":"Cosmos3CausalAttention shards query heads across tensor-parallel ranks. num_attention_heads must be divisible by the tensor-parallel world size (tp_size from get_tp_world_size()) so each rank gets an equal head count; otherwise it raises ValueError at construction.","triggerScenarios":"Launching Cosmos3 video generation with --tp N where N does not divide the model's num_attention_heads (e.g. tp=4 with 30 heads). The attention module __init__ raises immediately during model build.","commonSituations":"Choosing a TP degree tuned for a text LLM (e.g. tp=8) and reusing it for a video DiT with a different head count; using head counts that are products of small primes incompatible with the chosen TP.","solutions":["Pick a tp_size that divides num_attention_heads (any power of 2 up to the largest power of 2 dividing it, e.g. 1/2/4/8 for 64 heads)","Check the model's config for num_attention_heads before choosing --tp","Fall back to tp=1 if unsure or for small models"],"exampleFix":"# before\npython -m sglang.launch_server --model cosmos3 ... --tp 6   # 64 heads\n# after\npython -m sglang.launch_server --model cosmos3 ... --tp 4   # 64 % 4 == 0","handlingStrategy":"validation","validationCode":"tp = get_tp_world_size()\nassert num_attention_heads % tp == 0, f\"{num_attention_heads=} not divisible by tp={tp}\"","typeGuard":"def tp_divides(num_heads: int, tp: int) -> bool:\n    return num_heads % tp == 0","tryCatchPattern":null,"preventionTips":["Compute the largest power of 2 dividing num_attention_heads before choosing --tp","Automate a pre-launch config check that validates head counts against TP"],"tags":["sglang","cosmos3","tensor-parallel","attention-heads","config"],"backgroundTag":"attention-heads-not-divisible-by-tp","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}