sgl-project/sglang · error · ValueError

RoPE config mismatch across layers for fused KV path: expect

Error message

RoPE config mismatch across layers for fused KV path: expected (rotary_dim={self.rotary_dim}, neox={self.is_neox_style}), got (rotary_dim={layer_rotary_dim}, neox={layer_is_neox}) at layer {layer_id}.

What it means

Every layer's rotary_emb must have the same rotary_dim and is_neox_style as the reference config captured in __init__, because one shared cos/sin cache and kernel layout serve all layers.

Source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:316

            if int(attn.num_kv_heads) != self.num_kv_heads:
                raise ValueError(
                    "num_kv_heads mismatch across layers for fused KV path: "
                    f"expected {self.num_kv_heads}, got {int(attn.num_kv_heads)} at layer {layer_id}."
                )
            if int(attn.head_dim) != self.head_dim:
                raise ValueError(
                    "head_dim mismatch across layers for fused KV path: "
                    f"expected {self.head_dim}, got {int(attn.head_dim)} at layer {layer_id}."
                )
            layer_rotary_dim = int(
                getattr(attn.rotary_emb, "rotary_dim", self.head_dim)
            )
            layer_is_neox = bool(getattr(attn.rotary_emb, "is_neox_style", True))
            if (
                layer_rotary_dim != self.rotary_dim
                or layer_is_neox != self.is_neox_style
            ):
                raise ValueError(
                    "RoPE config mismatch across layers for fused KV path: "
                    f"expected (rotary_dim={self.rotary_dim}, neox={self.is_neox_style}), "
                    f"got (rotary_dim={layer_rotary_dim}, neox={layer_is_neox}) at layer {layer_id}."
                )

            qkv_w = attn.qkv_proj.weight
            kv_weight = qkv_w[attn.q_size : attn.q_size + 2 * attn.kv_size]
            kv_weights.append(kv_weight)
            k_norm_weights.append(attn.k_norm.weight)
            eps_values.append(float(attn.k_norm.variance_epsilon))

        flat_kv_weight = torch.stack(kv_weights).reshape(
            self.n_layers * self.layer_out_dim, -1
        )
        self.flat_kv_weight_t = flat_kv_weight.transpose(0, 1).contiguous()
        self.k_norm_weights = torch.stack(k_norm_weights).contiguous()
        self.eps_values = torch.tensor(
            eps_values, dtype=torch.float32, device=self.device

View on GitHub (pinned to 0132848349)

Solutions

Make rotary configs uniform across layers or disable the fused KV path.
Check getattr defaults: layers whose rotary_emb lacks rotary_dim default to head_dim — set the attribute explicitly if partial.
Diff each layer's rotary_emb.rotary_dim/is_neox_style during model init to catch divergence early.

Defensive patterns

Strategy: validation

Validate before calling

cfgs = {(int(getattr(l.self_attn.rotary_emb,'rotary_dim',hd)), bool(getattr(l.self_attn.rotary_emb,'is_neox_style',True))) for l in layers}
assert len(cfgs) == 1

Prevention

Set rotary_dim explicitly on every layer's rotary_emb when partial rotary is used.

When it happens

Trigger: A model where one layer's rotary_emb has a different rotary_dim or neox flag (hybrid RoPE configs, partial-rotary on only some layers).

Common situations: Checkpoints with mixed full/partial rotary (e.g. some search-optimized variants), or custom per-layer rotary_emb construction with inconsistent attributes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a9963382f08165b8. Report an issue: GitHub.