{"record":{"id":"d21b44486bc69a3c","repo":"sgl-project/sglang","slug":"num-kv-heads-mismatch-across-layers-for-fused-kv-p","errorCode":null,"errorMessage":"num_kv_heads mismatch across layers for fused KV path: expected {self.num_kv_heads}, got {int(attn.num_kv_heads)} at layer {layer_id}.","messagePattern":"num_kv_heads mismatch across layers for fused KV path: expected (.+?), got (.+?) at layer (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/speculative/fused_kv_materialize.py","lineNumber":299,"sourceCode":"        )\n        self._reserved_rope_cache_len = int(\n            getattr(self.rotary_emb, \"cos_sin_cache\", torch.empty((0,))).shape[0]\n        )\n        self._mm_out_supported = True\n        self._workspace_capacity = 0\n        self._workspace_dtype: Optional[torch.dtype] = None\n        self._proj_workspace: Optional[torch.Tensor] = None\n        self._k_workspace: Optional[torch.Tensor] = None\n        self._v_workspace: Optional[torch.Tensor] = None\n\n        kv_weights = []\n        k_norm_weights = []\n        eps_values = []\n\n        for layer_id, layer in enumerate(layers):\n            attn = layer.self_attn\n            if int(attn.num_kv_heads) != self.num_kv_heads:\n                raise ValueError(\n                    \"num_kv_heads mismatch across layers for fused KV path: \"\n                    f\"expected {self.num_kv_heads}, got {int(attn.num_kv_heads)} at layer {layer_id}.\"\n                )\n            if int(attn.head_dim) != self.head_dim:\n                raise ValueError(\n                    \"head_dim mismatch across layers for fused KV path: \"\n                    f\"expected {self.head_dim}, got {int(attn.head_dim)} at layer {layer_id}.\"\n                )\n            layer_rotary_dim = int(\n                getattr(attn.rotary_emb, \"rotary_dim\", self.head_dim)\n            )\n            layer_is_neox = bool(getattr(attn.rotary_emb, \"is_neox_style\", True))\n            if (\n                layer_rotary_dim != self.rotary_dim\n                or layer_is_neox != self.is_neox_style\n            ):\n                raise ValueError(\n                    \"RoPE config mismatch across layers for fused KV path: \"","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/speculative/fused_kv_materialize.py#L281-L317","documentation":"All decoder layers must have the same num_kv_heads for the stacked fused KV path, since one Triton launch handles all layers with a single head-count grid. The check compares each layer's attn.num_kv_heads against the first layer's.","triggerScenarios":"A model where some attention layers use different num_kv_heads (e.g. alternating GQA group sizes or layer-wise heterogeneous attention).","commonSituations":"New hybrid/checkpoint architectures with per-layer attention config arrays; passing a subset of layers with mismatched configs.","solutions":["Only enable the fused KV path on models with uniform num_kv_heads across layers.","Verify each layer's config in the checkpoint; fix any wrong per-layer overrides.","Fall back to the non-fused per-layer materialization for heterogeneous models."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"heads = {int(l.self_attn.num_kv_heads) for l in layers}\nassert len(heads) == 1","typeGuard":"def uniform_kv_heads(layers) -> bool:\n    return len({int(l.self_attn.num_kv_heads) for l in layers}) == 1","tryCatchPattern":null,"preventionTips":["Validate per-layer attention configs before enabling the fused speculative path."],"tags":["config-validation","gqa","speculative-decoding"],"backgroundTag":"config-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}