sgl-project/sglang · error · ValueError

hidden_size must be divisible by num_heads (got `hidden_size

Error message

hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size} and `num_heads`: {self.num_heads}).

What it means

StableLM attention verifies hidden_size == head_dim*num_heads*tp_size after TP division; if the per-rank head count doesn't tile the hidden size exactly, QKV projections would be misaligned and init aborts with the actual values.

Source

Thrown at python/sglang/srt/models/stablelm.py:123

            # the KV heads across multiple tensor parallel GPUs.
            assert self.total_num_key_value_heads % tp_size == 0
        else:
            # Number of KV heads is less than TP size, so we replicate
            # the KV heads across multiple tensor parallel GPUs.
            assert tp_size % self.total_num_key_value_heads == 0
        self.num_key_value_heads = max(1, self.total_num_key_value_heads // tp_size)
        self.head_dim = self.hidden_size // self.total_num_heads
        self.max_position_embeddings = config.max_position_embeddings
        rope_pct = getattr(
            config, "rope_pct", getattr(config, "partial_rotary_factor", 1)
        )
        self.rotary_ndims = int(self.head_dim * rope_pct)
        self.scaling = self.head_dim**-0.5
        self.q_size = self.num_heads * self.head_dim
        self.kv_size = self.num_key_value_heads * self.head_dim
        self.qkv_bias = getattr(config, "use_qkv_bias", False)
        if (self.head_dim * self.num_heads * tp_size) != self.hidden_size:
            raise ValueError(
                f"hidden_size must be divisible by num_heads "
                f"(got `hidden_size`: {self.hidden_size}"
                f" and `num_heads`: {self.num_heads})."
            )

        self.qkv_proj = QKVParallelLinear(
            self.hidden_size,
            self.head_dim,
            self.total_num_heads,
            self.total_num_key_value_heads,
            self.qkv_bias,
            quant_config=quant_config,
            prefix=add_prefix("qkv_proj", prefix),
        )
        self.o_proj = RowParallelLinear(
            self.total_num_heads * self.head_dim,
            self.hidden_size,
            bias=False,

View on GitHub (pinned to 0132848349)

Solutions

1. Choose --tp-size that divides num_attention_heads (1, 2, 4, ... common divisors)
2. Verify config.json hidden_size == head_dim * num_attention_heads
3. For odd TP requirements, prefer DP + small TP instead

Example fix

# before
--tp-size 6   # 16 heads not divisible
# after
--tp-size 4

assert config.hidden_size % (config.num_attention_heads) == 0
assert config.num_attention_heads % tp_size == 0

When it happens

Trigger: Launching a StableLM model with --tp-size that doesn't divide num_heads (e.g. StableLM-head-32 with tp=3), or a config.json with inconsistent hidden_size/num_heads (including num_key_value_heads-derived head_dim).

Common situations: Scripts tuned for models whose head count is a power of two reused on StableLM (e.g. 16 heads with tp=6); 4-bit/8-bit conversions that mangle config fields.

Related errors

• out_channels must be divisible by tp_size for TP-sharded out
• MiniMax H3 {name}={value} must be divisible by TP size {tp_s
• MiniMax H3 TP-local heads {local_heads} must be divisible by
• num_heads ({self.num_heads}) must be divisible by tp_size ({
• tensor_model_parallel_size ({tensor_model_parallel_size}) mu

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/13479ad0bd72c55e. Report an issue: GitHub.