{"record":{"id":"310d1680dca613e6","repo":"sgl-project/sglang","slug":"ltx2attention-requires-inner-dim-divisible-by-tp-s","errorCode":null,"errorMessage":"LTX2Attention requires inner_dim divisible by tp_size, got {self.inner_dim=} {tp_size=}.","messagePattern":"LTX2Attention requires inner_dim divisible by tp_size, got (.+?) (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py","lineNumber":766,"sourceCode":"        self.inner_dim = self.heads * self.dim_head\n        self.norm_eps = float(norm_eps)\n        self.qk_norm = bool(qk_norm)\n        self.use_local_attention = bool(use_local_attention)\n        self.apply_gated_attention = bool(apply_gated_attention)\n        self.enable_packed_qkv_input_a2a = bool(enable_packed_qkv_input_a2a)\n        self.prefix = prefix\n\n        tp_size = get_tp_world_size()\n        if tp_size <= 0:\n            raise ValueError(f\"Invalid {tp_size=}. Expected tp_size >= 1.\")\n        if self.heads % tp_size != 0:\n            raise ValueError(\n                f\"LTX2Attention requires heads divisible by tp_size, got \"\n                f\"{self.heads=} {tp_size=}.\"\n            )\n        if self.inner_dim % tp_size != 0:\n            # This should follow from heads % tp_size, but keep explicit for clarity.\n            raise ValueError(\n                f\"LTX2Attention requires inner_dim divisible by tp_size, got \"\n                f\"{self.inner_dim=} {tp_size=}.\"\n            )\n        self.local_heads = self.heads // tp_size\n\n        self.to_q = ColumnParallelLinear(\n            self.query_dim,\n            self.inner_dim,\n            bias=True,\n            gather_output=False,\n            quant_config=quant_config,\n        )\n        self.to_k = ColumnParallelLinear(\n            self.context_dim,\n            self.inner_dim,\n            bias=True,\n            gather_output=False,\n            quant_config=quant_config,","sourceCodeStart":748,"sourceCodeEnd":784,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py#L748-L784","documentation":"LTX2Attention also checks that the total inner dimension (heads * head_dim) divides across TP ranks. This normally follows from heads % tp_size, but is kept explicit to catch configs where head_dim itself is not divisible by tp_size, which would break the ColumnParallelLinear output split.","triggerScenarios":"inner_dim (e.g. heads*head_dim = 24*40 = 960) not divisible by tp_size, while heads happens to divide — possible only with unusual head_dim values relative to the TP degree.","commonSituations":"Custom head_dim configs (non-64/128 dims like 40 or 80) combined with high TP degrees; importing dims from a different attention implementation.","solutions":["Pick tp_size that divides both heads and heads*head_dim","Use a standard head_dim (64, 128) which divides cleanly for power-of-two TP","Adjust head_dim in the config if the checkpoint allows"],"exampleFix":"# before\n--tp-size 8   # inner_dim=960 -> 960%8 != 0\n\n# after\n--tp-size 4   # 960/4 = 240 per rank","handlingStrategy":"validation","validationCode":"tp = get_tp_world_size()\\ninner = config.num_attention_heads * config.head_dim\\nassert inner % tp == 0, f'inner_dim {inner} % tp {tp} != 0'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check both heads and heads*head_dim divisibility before launch","Use standard head_dims (64/128)","Automate the parallel-layout precheck in CI"],"tags":["parallelism","tensor-parallel","inner-dim","config-validation","ltx2"],"backgroundTag":"parallel-degree-not-divisible","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}