{"record":{"id":"13479ad0bd72c55e","repo":"sgl-project/sglang","slug":"hidden-size-must-be-divisible-by-num-heads-got-h-13479a","errorCode":null,"errorMessage":"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size} and `num_heads`: {self.num_heads}).","messagePattern":"hidden_size must be divisible by num_heads \\(got `hidden_size`: (.+?) and `num_heads`: (.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/stablelm.py","lineNumber":123,"sourceCode":"            # the KV heads across multiple tensor parallel GPUs.\n            assert self.total_num_key_value_heads % tp_size == 0\n        else:\n            # Number of KV heads is less than TP size, so we replicate\n            # the KV heads across multiple tensor parallel GPUs.\n            assert tp_size % self.total_num_key_value_heads == 0\n        self.num_key_value_heads = max(1, self.total_num_key_value_heads // tp_size)\n        self.head_dim = self.hidden_size // self.total_num_heads\n        self.max_position_embeddings = config.max_position_embeddings\n        rope_pct = getattr(\n            config, \"rope_pct\", getattr(config, \"partial_rotary_factor\", 1)\n        )\n        self.rotary_ndims = int(self.head_dim * rope_pct)\n        self.scaling = self.head_dim**-0.5\n        self.q_size = self.num_heads * self.head_dim\n        self.kv_size = self.num_key_value_heads * self.head_dim\n        self.qkv_bias = getattr(config, \"use_qkv_bias\", False)\n        if (self.head_dim * self.num_heads * tp_size) != self.hidden_size:\n            raise ValueError(\n                f\"hidden_size must be divisible by num_heads \"\n                f\"(got `hidden_size`: {self.hidden_size}\"\n                f\" and `num_heads`: {self.num_heads}).\"\n            )\n\n        self.qkv_proj = QKVParallelLinear(\n            self.hidden_size,\n            self.head_dim,\n            self.total_num_heads,\n            self.total_num_key_value_heads,\n            self.qkv_bias,\n            quant_config=quant_config,\n            prefix=add_prefix(\"qkv_proj\", prefix),\n        )\n        self.o_proj = RowParallelLinear(\n            self.total_num_heads * self.head_dim,\n            self.hidden_size,\n            bias=False,","sourceCodeStart":105,"sourceCodeEnd":141,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/stablelm.py#L105-L141","documentation":"StableLM attention verifies hidden_size == head_dim*num_heads*tp_size after TP division; if the per-rank head count doesn't tile the hidden size exactly, QKV projections would be misaligned and init aborts with the actual values.","triggerScenarios":"Launching a StableLM model with --tp-size that doesn't divide num_heads (e.g. StableLM-head-32 with tp=3), or a config.json with inconsistent hidden_size/num_heads (including num_key_value_heads-derived head_dim).","commonSituations":"Scripts tuned for models whose head count is a power of two reused on StableLM (e.g. 16 heads with tp=6); 4-bit/8-bit conversions that mangle config fields.","solutions":["Choose --tp-size that divides num_attention_heads (1, 2, 4, ... common divisors)","Verify config.json hidden_size == head_dim * num_attention_heads","For odd TP requirements, prefer DP + small TP instead"],"exampleFix":"# before\n--tp-size 6   # 16 heads not divisible\n# after\n--tp-size 4","handlingStrategy":"validation","validationCode":"assert config.hidden_size % (config.num_attention_heads) == 0\nassert config.num_attention_heads % tp_size == 0","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pick TP sizes from the divisors of num_attention_heads"],"tags":["stablelm","tensor-parallel","divisibility"],"backgroundTag":"tp-head-divisibility","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}