{"record":{"id":"2916405633ddca1a","repo":"sgl-project/sglang","slug":"cosmos3crossattention-requires-num-attention-heads","errorCode":null,"errorMessage":"Cosmos3CrossAttention requires num_attention_heads divisible by tp_size, got {num_attention_heads=} {self.tp_size=}.","messagePattern":"Cosmos3CrossAttention requires num_attention_heads divisible by tp_size, got (.+?) (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/cosmos3video.py","lineNumber":721,"sourceCode":"\n    def __init__(\n        self,\n        hidden_size: int,\n        num_attention_heads: int,\n        num_key_value_heads: int,\n        head_dim: int,\n        prefix: str = \"\",\n        quant_config: QuantizationConfig | None = None,\n        supported_attention_backends: set | None = None,\n    ):\n        super().__init__()\n        self.hidden_size = hidden_size\n        self.num_attention_heads = num_attention_heads\n        self.num_key_value_heads = num_key_value_heads\n        self.head_dim = head_dim\n        self.tp_size = get_tp_world_size()\n        if num_attention_heads % self.tp_size != 0:\n            raise ValueError(\n                \"Cosmos3CrossAttention requires num_attention_heads divisible \"\n                f\"by tp_size, got {num_attention_heads=} {self.tp_size=}.\"\n            )\n        if num_key_value_heads % self.tp_size != 0:\n            raise ValueError(\n                \"Cosmos3CrossAttention requires num_key_value_heads divisible \"\n                f\"by tp_size, got {num_key_value_heads=} {self.tp_size=}.\"\n            )\n        self.local_num_attention_heads = num_attention_heads // self.tp_size\n        self.local_num_key_value_heads = num_key_value_heads // self.tp_size\n\n        self.q_size = num_attention_heads * head_dim\n        self.kv_size = num_key_value_heads * head_dim\n        self.to_qkv = MergedColumnParallelLinear(\n            hidden_size,\n            [self.q_size, self.kv_size, self.kv_size],\n            bias=False,\n            gather_output=False,","sourceCodeStart":703,"sourceCodeEnd":739,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/cosmos3video.py#L703-L739","documentation":"The cross-attention block in Cosmos3 shards text-conditioning query heads across tensor-parallel ranks. If num_attention_heads of the cross-attention layer is not divisible by tp_size, __init__ raises this ValueError during model build.","triggerScenarios":"Launching Cosmos3 with a TP degree that divides the self-attention head count but not the cross-attention head count (some configs use different head counts for cross-attention), e.g. tp=8 with 12 cross-attn heads.","commonSituations":"Configs where cross_attention layers have distinct num_attention_heads from causal layers; validating TP only against the main attention config and missing the cross-attention head count.","solutions":["Check the cross-attention num_attention_heads in the model config and pick a --tp that divides it (as well as the causal-layer head counts)","Use a smaller TP that satisfies all head-count divisibility constraints","Add a startup assertion script that validates all three head counts against your chosen tp before launching"],"exampleFix":"# before\n--tp 6   # cross-attn heads = 8\n# after\n--tp 4   # 8 % 4 == 0","handlingStrategy":"validation","validationCode":"for name in (\"num_attention_heads\", \"num_key_value_heads\"):\n    for sub in (\"self_attn\", \"cross_attn\"):\n        assert cfg[sub][name] % tp == 0, f\"{sub}.{name} not divisible by tp={tp}\"","typeGuard":"def tp_valid_for_config(cfg, tp: int) -> bool:\n    return all(h % tp == 0 for h in all_head_counts(cfg))","tryCatchPattern":null,"preventionTips":["Validate every attention submodule's head counts, not just the main one","Add a startup script that loads the config and checks all divisibility constraints"],"tags":["sglang","cosmos3","tensor-parallel","cross-attention"],"backgroundTag":"attention-heads-not-divisible-by-tp","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}