sgl-project/sglang · error · ValueError

num_attention_heads must be divisible by attention TP

Error message

num_attention_heads must be divisible by attention TP

What it means

The attention module validates that total attention heads divide evenly across attention tensor-parallel ranks. If config.num_attention_heads % attn_tp_size != 0, each rank cannot receive an equal shard of heads.

Source

Thrown at python/sglang/srt/models/interns2_mobius.py:566

):
    """Mobius-owned full-attention constructor reusing Qwen3.5 methods."""

    def __init__(
        self,
        config: InternS2MobiusTextConfig,
        layer_id: int,
        quant_config: QuantizationConfig | None = None,
        prefix: str = "",
        alt_stream: torch.cuda.Stream | None = None,
    ) -> None:
        nn.Module.__init__(self)
        self.config = config
        self.hidden_size = config.hidden_size
        self.attn_tp_rank = get_parallel().attn_tp_rank
        self.attn_tp_size = get_parallel().attn_tp_size
        self.total_num_heads = config.num_attention_heads
        if self.total_num_heads % self.attn_tp_size != 0:
            raise ValueError("num_attention_heads must be divisible by attention TP")
        self.num_heads = self.total_num_heads // self.attn_tp_size
        self.total_num_kv_heads = config.num_key_value_heads
        if self.total_num_kv_heads >= self.attn_tp_size:
            if self.total_num_kv_heads % self.attn_tp_size != 0:
                raise ValueError(
                    "num_key_value_heads must be divisible by attention TP"
                )
        elif self.attn_tp_size % self.total_num_kv_heads != 0:
            raise ValueError("attention TP must be divisible by num_key_value_heads")
        self.num_kv_heads = max(1, self.total_num_kv_heads // self.attn_tp_size)
        self.head_dim = config.head_dim or (self.hidden_size // self.num_heads)
        self.q_size = self.num_heads * self.head_dim
        self.kv_size = self.num_kv_heads * self.head_dim
        self.scaling = self.head_dim**-0.5
        self.max_position_embeddings = getattr(config, "max_position_embeddings", 8192)
        self.rope_theta, rope_scaling = get_rope_config(config)
        self.partial_rotary_factor = getattr(config, "partial_rotary_factor", 1.0)
        self.layer_id = layer_id

View on GitHub (pinned to 0132848349)

Solutions

1. Lower or change TP size so it divides num_attention_heads (e.g., use tp=4 instead of 8 for 40 heads)
2. Check effective attn_tp_size via get_parallel() with your dp-attention flags
3. Confirm the checkpoint's config.json num_attention_heads matches the released model

Example fix

# before
python -m sglang.launch_server --model intern-s2-mobius --tp 8   # 40 heads

# after
python -m sglang.launch_server --model intern-s2-mobius --tp 5   # 40/5=8 heads per rank

assert config.num_attention_heads % get_parallel().attn_tp_size == 0

def heads_divisible_by_attn_tp(config, attn_tp_size: int) -> bool:
    return config.num_attention_heads % attn_tp_size == 0

When it happens

Trigger: Constructing the decoder layer with attn_tp_size that does not divide config.num_attention_heads — e.g., 40 heads with --tp-size 8 and attention TP decomposition routing to a non-divisor size.

Common situations: Launching with an aggressive TP degree for a small model, using --enable-dp-attention or --dp-size which changes attn_tp_size, or mis-set default config in a fine-tuned variant.

Related errors

• Tensor parallel size {self.tp_size} is greater than the numb
• Tensor parallel size {self.tp_size} > num_experts {config.nu
• delta payload size mismatch: expected ${expectedSize}, got $
• This browser does not support worker image decoding
• Generate subcommand is not yet supported for model: {model_p

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/dd4243aaaafe259f. Report an issue: GitHub.