sgl-project/sglang · error · ValueError

attention TP must be divisible by num_key_value_heads

Error message

attention TP must be divisible by num_key_value_heads

What it means

When attn_tp_size > total_num_kv_heads, each KV head must be replicated across an integer number of ranks: attn_tp_size % num_key_value_heads == 0. Otherwise some ranks would own fractional KV heads.

Source

Thrown at python/sglang/srt/models/interns2_mobius.py:575

        alt_stream: torch.cuda.Stream | None = None,
    ) -> None:
        nn.Module.__init__(self)
        self.config = config
        self.hidden_size = config.hidden_size
        self.attn_tp_rank = get_parallel().attn_tp_rank
        self.attn_tp_size = get_parallel().attn_tp_size
        self.total_num_heads = config.num_attention_heads
        if self.total_num_heads % self.attn_tp_size != 0:
            raise ValueError("num_attention_heads must be divisible by attention TP")
        self.num_heads = self.total_num_heads // self.attn_tp_size
        self.total_num_kv_heads = config.num_key_value_heads
        if self.total_num_kv_heads >= self.attn_tp_size:
            if self.total_num_kv_heads % self.attn_tp_size != 0:
                raise ValueError(
                    "num_key_value_heads must be divisible by attention TP"
                )
        elif self.attn_tp_size % self.total_num_kv_heads != 0:
            raise ValueError("attention TP must be divisible by num_key_value_heads")
        self.num_kv_heads = max(1, self.total_num_kv_heads // self.attn_tp_size)
        self.head_dim = config.head_dim or (self.hidden_size // self.num_heads)
        self.q_size = self.num_heads * self.head_dim
        self.kv_size = self.num_kv_heads * self.head_dim
        self.scaling = self.head_dim**-0.5
        self.max_position_embeddings = getattr(config, "max_position_embeddings", 8192)
        self.rope_theta, rope_scaling = get_rope_config(config)
        self.partial_rotary_factor = getattr(config, "partial_rotary_factor", 1.0)
        self.layer_id = layer_id
        if rope_scaling and not ("rope_type" in rope_scaling or "type" in rope_scaling):
            rope_scaling = None
        self.attn_output_gate = getattr(config, "attn_output_gate", True)
        self.rotary_emb = get_rope(
            head_size=self.head_dim,
            rotary_dim=self.head_dim,
            max_position=self.max_position_embeddings,
            rope_scaling=rope_scaling,
            base=self.rope_theta,

View on GitHub (pinned to 0132848349)

Solutions

1. Pick TP size that is a multiple of num_key_value_heads (tp=3 or 6 for 3 KV heads)
2. Use --dp-size to scale down attn_tp_size to a divisor/multiple relationship
3. Serve with lower TP such that attn_tp_size <= num_key_value_heads and divides it

Example fix

# before
--tp 8   # 3 kv heads: 8 % 3 != 0

# after
--tp 3    # exact replication factor 1

kv, tp = config.num_key_value_heads, get_parallel().attn_tp_size
assert kv >= tp or tp % kv == 0

def kv_replication_ok(kv: int, tp: int) -> bool:
    return tp % kv == 0

When it happens

Trigger: e.g., 3 KV heads with attn_tp_size 8 (8 % 3 != 0) — replication factor would be non-integral.

Common situations: Deep GQA models (very few KV heads) under large TP degrees; enabling dp-attention which shrinks attn_tp_size can inadvertently avoid or trigger this.

Related errors

• Cosmos3CausalAttention requires num_key_value_heads divisibl
• num_key_value_heads must be divisible by attention TP
• Cosmos3CrossAttention requires num_key_value_heads divisible
• qkv_proj weight {name}: unexpected shape {tuple(loaded_weigh
• delta payload size mismatch: expected ${expectedSize}, got $

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7c4923d1e22e8dfe. Report an issue: GitHub.