sgl-project/sglang · critical · ValueError

LTX2Attention requires heads divisible by tp_size, got {self

Error message

LTX2Attention requires heads divisible by tp_size, got {self.heads=} {tp_size=}.

What it means

Under tensor parallelism each rank holds heads/tp_size heads, so the total head count must be divisible by tp_size. LTX2Attention checks this at init right after resolving the TP world size.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py:760

        is_cross_attention = context_dim is not None
        self.query_dim = int(query_dim)
        self.context_dim = int(query_dim if context_dim is None else context_dim)
        self.heads = int(heads)
        self.dim_head = int(dim_head)
        self.inner_dim = self.heads * self.dim_head
        self.norm_eps = float(norm_eps)
        self.qk_norm = bool(qk_norm)
        self.use_local_attention = bool(use_local_attention)
        self.apply_gated_attention = bool(apply_gated_attention)
        self.enable_packed_qkv_input_a2a = bool(enable_packed_qkv_input_a2a)
        self.prefix = prefix

        tp_size = get_tp_world_size()
        if tp_size <= 0:
            raise ValueError(f"Invalid {tp_size=}. Expected tp_size >= 1.")
        if self.heads % tp_size != 0:
            raise ValueError(
                f"LTX2Attention requires heads divisible by tp_size, got "
                f"{self.heads=} {tp_size=}."
            )
        if self.inner_dim % tp_size != 0:
            # This should follow from heads % tp_size, but keep explicit for clarity.
            raise ValueError(
                f"LTX2Attention requires inner_dim divisible by tp_size, got "
                f"{self.inner_dim=} {tp_size=}."
            )
        self.local_heads = self.heads // tp_size

        self.to_q = ColumnParallelLinear(
            self.query_dim,
            self.inner_dim,
            bias=True,
            gather_output=False,
            quant_config=quant_config,
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Choose a tp_size that divides the head count (factors of heads)
2. Reduce tp_size (e.g. to 1, 2, or the largest divisor available)
3. If you control the config, pick a head count friendly to your TP layout (multiples of 8/tp)

Example fix

# before
--tp-size 8   # heads=30 -> 30%8 != 0

# after
--tp-size 5   # heads=30 -> 6 heads per rank

tp = get_tp_world_size()\nassert config.num_attention_heads % tp == 0, f'heads {config.num_attention_heads} % tp {tp} != 0'

When it happens

Trigger: Launching with --tp-size that does not divide config heads, e.g. heads=24 with tp_size=8 gives 3 heads/rank fine, but heads=30 with tp_size=8 (30%8!=0) fails.

Common situations: Scaling TP up to 8 GPUs on a small model; swapping checkpoints with unusual head counts; combining TP with other parallel dims that change effective heads.

Related errors

• LTX2Attention requires inner_dim divisible by tp_size, got {
• num_heads ({self.num_heads}) must be divisible by ulysses_de
• Invalid {tp_size=}. Expected tp_size >= 1.
• world_size ({world_size}) is not equal to tensor_model_paral
• tensor_model_parallel_size ({tensor_model_parallel_size}) mu

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bdc8183dac1e0d96. Report an issue: GitHub.