sgl-project/sglang · critical · ValueError

LTX2Attention requires inner_dim divisible by tp_size, got {

Error message

LTX2Attention requires inner_dim divisible by tp_size, got {self.inner_dim=} {tp_size=}.

What it means

LTX2Attention also checks that the total inner dimension (heads * head_dim) divides across TP ranks. This normally follows from heads % tp_size, but is kept explicit to catch configs where head_dim itself is not divisible by tp_size, which would break the ColumnParallelLinear output split.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py:766

        self.inner_dim = self.heads * self.dim_head
        self.norm_eps = float(norm_eps)
        self.qk_norm = bool(qk_norm)
        self.use_local_attention = bool(use_local_attention)
        self.apply_gated_attention = bool(apply_gated_attention)
        self.enable_packed_qkv_input_a2a = bool(enable_packed_qkv_input_a2a)
        self.prefix = prefix

        tp_size = get_tp_world_size()
        if tp_size <= 0:
            raise ValueError(f"Invalid {tp_size=}. Expected tp_size >= 1.")
        if self.heads % tp_size != 0:
            raise ValueError(
                f"LTX2Attention requires heads divisible by tp_size, got "
                f"{self.heads=} {tp_size=}."
            )
        if self.inner_dim % tp_size != 0:
            # This should follow from heads % tp_size, but keep explicit for clarity.
            raise ValueError(
                f"LTX2Attention requires inner_dim divisible by tp_size, got "
                f"{self.inner_dim=} {tp_size=}."
            )
        self.local_heads = self.heads // tp_size

        self.to_q = ColumnParallelLinear(
            self.query_dim,
            self.inner_dim,
            bias=True,
            gather_output=False,
            quant_config=quant_config,
        )
        self.to_k = ColumnParallelLinear(
            self.context_dim,
            self.inner_dim,
            bias=True,
            gather_output=False,
            quant_config=quant_config,

View on GitHub (pinned to 0132848349)

Solutions

1. Pick tp_size that divides both heads and heads*head_dim
2. Use a standard head_dim (64, 128) which divides cleanly for power-of-two TP
3. Adjust head_dim in the config if the checkpoint allows

Example fix

# before
--tp-size 8   # inner_dim=960 -> 960%8 != 0

# after
--tp-size 4   # 960/4 = 240 per rank

tp = get_tp_world_size()\ninner = config.num_attention_heads * config.head_dim\nassert inner % tp == 0, f'inner_dim {inner} % tp {tp} != 0'

When it happens

Trigger: inner_dim (e.g. heads*head_dim = 24*40 = 960) not divisible by tp_size, while heads happens to divide — possible only with unusual head_dim values relative to the TP degree.

Common situations: Custom head_dim configs (non-64/128 dims like 40 or 80) combined with high TP degrees; importing dims from a different attention implementation.

Related errors

• LTX2Attention requires heads divisible by tp_size, got {self
• Invalid {tp_size=}. Expected tp_size >= 1.
• world_size ({world_size}) is not equal to tensor_model_paral
• tensor_model_parallel_size ({tensor_model_parallel_size}) mu
• num_heads ({self.num_heads}) must be divisible by ulysses_de

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/310d1680dca613e6. Report an issue: GitHub.