{"record":{"id":"bdc8183dac1e0d96","repo":"sgl-project/sglang","slug":"ltx2attention-requires-heads-divisible-by-tp-size","errorCode":null,"errorMessage":"LTX2Attention requires heads divisible by tp_size, got {self.heads=} {tp_size=}.","messagePattern":"LTX2Attention requires heads divisible by tp_size, got (.+?) (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py","lineNumber":760,"sourceCode":"\n        is_cross_attention = context_dim is not None\n        self.query_dim = int(query_dim)\n        self.context_dim = int(query_dim if context_dim is None else context_dim)\n        self.heads = int(heads)\n        self.dim_head = int(dim_head)\n        self.inner_dim = self.heads * self.dim_head\n        self.norm_eps = float(norm_eps)\n        self.qk_norm = bool(qk_norm)\n        self.use_local_attention = bool(use_local_attention)\n        self.apply_gated_attention = bool(apply_gated_attention)\n        self.enable_packed_qkv_input_a2a = bool(enable_packed_qkv_input_a2a)\n        self.prefix = prefix\n\n        tp_size = get_tp_world_size()\n        if tp_size <= 0:\n            raise ValueError(f\"Invalid {tp_size=}. Expected tp_size >= 1.\")\n        if self.heads % tp_size != 0:\n            raise ValueError(\n                f\"LTX2Attention requires heads divisible by tp_size, got \"\n                f\"{self.heads=} {tp_size=}.\"\n            )\n        if self.inner_dim % tp_size != 0:\n            # This should follow from heads % tp_size, but keep explicit for clarity.\n            raise ValueError(\n                f\"LTX2Attention requires inner_dim divisible by tp_size, got \"\n                f\"{self.inner_dim=} {tp_size=}.\"\n            )\n        self.local_heads = self.heads // tp_size\n\n        self.to_q = ColumnParallelLinear(\n            self.query_dim,\n            self.inner_dim,\n            bias=True,\n            gather_output=False,\n            quant_config=quant_config,\n        )","sourceCodeStart":742,"sourceCodeEnd":778,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py#L742-L778","documentation":"Under tensor parallelism each rank holds heads/tp_size heads, so the total head count must be divisible by tp_size. LTX2Attention checks this at init right after resolving the TP world size.","triggerScenarios":"Launching with --tp-size that does not divide config heads, e.g. heads=24 with tp_size=8 gives 3 heads/rank fine, but heads=30 with tp_size=8 (30%8!=0) fails.","commonSituations":"Scaling TP up to 8 GPUs on a small model; swapping checkpoints with unusual head counts; combining TP with other parallel dims that change effective heads.","solutions":["Choose a tp_size that divides the head count (factors of heads)","Reduce tp_size (e.g. to 1, 2, or the largest divisor available)","If you control the config, pick a head count friendly to your TP layout (multiples of 8/tp)"],"exampleFix":"# before\n--tp-size 8   # heads=30 -> 30%8 != 0\n\n# after\n--tp-size 5   # heads=30 -> 6 heads per rank","handlingStrategy":"validation","validationCode":"tp = get_tp_world_size()\\nassert config.num_attention_heads % tp == 0, f'heads {config.num_attention_heads} % tp {tp} != 0'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Choose tp_size from the factors of the head count","Script a divisibility precheck from config + launch args","Prefer head counts divisible by 8 for TP flexibility"],"tags":["parallelism","tensor-parallel","attention-heads","config-validation","ltx2"],"backgroundTag":"parallel-degree-not-divisible","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}