sgl-project/sglang · error · ValueError

head_dim mismatch across layers for fused KV path: expected

Error message

head_dim mismatch across layers for fused KV path: expected {self.head_dim}, got {int(attn.head_dim)} at layer {layer_id}.

What it means

All layers must share the same head_dim for the fused KV path — the kernel's BLOCK_HD is derived from a single head_dim. Layer-level head_dim differing from the first layer's raises this.

Source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:304

        self._workspace_capacity = 0
        self._workspace_dtype: Optional[torch.dtype] = None
        self._proj_workspace: Optional[torch.Tensor] = None
        self._k_workspace: Optional[torch.Tensor] = None
        self._v_workspace: Optional[torch.Tensor] = None

        kv_weights = []
        k_norm_weights = []
        eps_values = []

        for layer_id, layer in enumerate(layers):
            attn = layer.self_attn
            if int(attn.num_kv_heads) != self.num_kv_heads:
                raise ValueError(
                    "num_kv_heads mismatch across layers for fused KV path: "
                    f"expected {self.num_kv_heads}, got {int(attn.num_kv_heads)} at layer {layer_id}."
                )
            if int(attn.head_dim) != self.head_dim:
                raise ValueError(
                    "head_dim mismatch across layers for fused KV path: "
                    f"expected {self.head_dim}, got {int(attn.head_dim)} at layer {layer_id}."
                )
            layer_rotary_dim = int(
                getattr(attn.rotary_emb, "rotary_dim", self.head_dim)
            )
            layer_is_neox = bool(getattr(attn.rotary_emb, "is_neox_style", True))
            if (
                layer_rotary_dim != self.rotary_dim
                or layer_is_neox != self.is_neox_style
            ):
                raise ValueError(
                    "RoPE config mismatch across layers for fused KV path: "
                    f"expected (rotary_dim={self.rotary_dim}, neox={self.is_neox_style}), "
                    f"got (rotary_dim={layer_rotary_dim}, neox={layer_is_neox}) at layer {layer_id}."
                )

            qkv_w = attn.qkv_proj.weight

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure uniform head_dim across all layers before enabling fused KV.
2. Re-check the model's per-layer config JSON / weights.
3. Use the fallback per-layer path for mixed head_dim models.

dims = {int(l.self_attn.head_dim) for l in layers}
assert len(dims) == 1

When it happens

Trigger: Models with mixed head dimensions across layers (some new hybrid architectures, misloaded per-layer configs).

Common situations: Per-layer attention config lists where head_dim entries differ, or a partially converted checkpoint.

Related errors

• Unsupported attention type: {config.attention_type}
• Unsupported attention type: {config.attention_type}
• num_heads must be divisible by num_epi_subtiles
• num_heads // num_epi_subtiles must be divisible by 4 (FMA un
• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0105de096d3a3506. Report an issue: GitHub.