sgl-project/sglang · error · ValueError
RoPE config mismatch across layers for fused KV path…
Error message
RoPE config mismatch across layers for fused KV path: expected (rotary_dim={self.rotary_dim}, neox={self.is_neox_style}), got (rotary_dim={layer_rotary_dim}, neox={layer_is_neox}) at layer {layer_id}. What it means
Cross-layer consistency check in fused KV materialization: every layer must use the same RoPE configuration (rotary_dim and neox style). A layer with a different RoPE config invalidates the shared cos/sin cache used by the fused kernel.
Solutions
- Make rotary configs uniform across layers or disable the fused KV path.
- Check getattr defaults: layers whose rotary_emb lacks rotary_dim default to head_dim — set the attribute explicitly if partial.
- Diff each layer's rotary_emb.rotary_dim/is_neox_style during model init to catch divergence early.
Defensive patterns
Strategy: validation
Validate before calling
cfgs = {(int(getattr(l.self_attn.rotary_emb,'rotary_dim',hd)), bool(getattr(l.self_attn.rotary_emb,'is_neox_style',True))) for l in layers}
assert len(cfgs) == 1 Prevention
- Set rotary_dim explicitly on every layer's rotary_emb when partial rotary is used.
When it happens
Trigger: A model where one layer's rotary_emb has a different rotary_dim or neox flag (hybrid RoPE configs, partial-rotary on only some layers).
Common situations: Checkpoints with mixed full/partial rotary (e.g. some search-optimized variants), or custom per-layer rotary_emb construction with inconsistent attributes.
Related errors
- Invalid fused KV rotary/head dim pair: rotary_dim=
- Invalid fused KV rotary/head dim pair: rotary_dim=
- memory_position_mode must be one of
- Missing rope_parameters
- no_rope_layers contains non-binary entries
AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28).
Data as JSON: /api/errors/a9963382f08165b8.
Report an issue: GitHub.
Appendix: source
Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:316
if int(attn.num_kv_heads) != self.num_kv_heads:
raise ValueError(
"num_kv_heads mismatch across layers for fused KV path: "
f"expected {self.num_kv_heads}, got {int(attn.num_kv_heads)} at layer {layer_id}."
)
if int(attn.head_dim) != self.head_dim:
raise ValueError(
"head_dim mismatch across layers for fused KV path: "
f"expected {self.head_dim}, got {int(attn.head_dim)} at layer {layer_id}."
)
layer_rotary_dim = int(
getattr(attn.rotary_emb, "rotary_dim", self.head_dim)
)
layer_is_neox = bool(getattr(attn.rotary_emb, "is_neox_style", True))
if (
layer_rotary_dim != self.rotary_dim
or layer_is_neox != self.is_neox_style
):
raise ValueError(
"RoPE config mismatch across layers for fused KV path: "
f"expected (rotary_dim={self.rotary_dim}, neox={self.is_neox_style}), "
f"got (rotary_dim={layer_rotary_dim}, neox={layer_is_neox}) at layer {layer_id}."
)
qkv_w = attn.qkv_proj.weight
kv_weight = qkv_w[attn.q_size : attn.q_size + 2 * attn.kv_size]
kv_weights.append(kv_weight)
k_norm_weights.append(attn.k_norm.weight)
eps_values.append(float(attn.k_norm.variance_epsilon))
flat_kv_weight = torch.stack(kv_weights).reshape(
self.n_layers * self.layer_out_dim, -1
)
self.flat_kv_weight_t = flat_kv_weight.transpose(0, 1).contiguous()
self.k_norm_weights = torch.stack(k_norm_weights).contiguous()
self.eps_values = torch.tensor(
eps_values, dtype=torch.float32, device=self.deviceView on GitHub (pinned to 0132848349)