sgl-project/sglang · error · ValueError

Invalid stacked eps shape for fused KV materialization: got

Error message

Invalid stacked eps shape for fused KV materialization: got {tuple(eps.shape)}, expected {(n_layers,)}.

What it means

The per-layer RMSNorm epsilon values must be supplied as a 1D tensor of length n_layers, one eps per layer. Any other shape is rejected.

Source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:169

    kv_size = num_kv_heads * head_dim
    if kv_dim != kv_size * 2:
        raise ValueError(
            "Invalid fused KV projection shape: "
            f"got {tuple(kv.shape)}, expected trailing dim {kv_size * 2}."
        )
    if rotary_dim <= 0 or rotary_dim > head_dim or rotary_dim % 2 != 0:
        raise ValueError(
            "Invalid fused KV rotary/head dim pair: "
            f"rotary_dim={rotary_dim}, head_dim={head_dim}."
        )
    if k_norm_weight.shape != (n_layers, head_dim):
        raise ValueError(
            "Invalid stacked k_norm_weight shape for fused KV materialization: "
            f"got {tuple(k_norm_weight.shape)}, expected {(n_layers, head_dim)}."
        )
    if eps.shape != (n_layers,):
        raise ValueError(
            "Invalid stacked eps shape for fused KV materialization: "
            f"got {tuple(eps.shape)}, expected {(n_layers,)}."
        )

    half_rotary_dim = rotary_dim // 2
    BLOCK_HD = triton.next_power_of_2(head_dim)

    if positions.device != kv.device:
        positions = positions.to(device=kv.device, dtype=torch.int64)
    elif positions.dtype != torch.int64:
        positions = positions.to(torch.int64)

    expected_shape = (n_layers, total_ctx, num_kv_heads, head_dim)
    if k_out is None:
        k_out = torch.empty(expected_shape, dtype=kv.dtype, device=kv.device)
    else:
        if k_out.shape != expected_shape:
            raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

Build eps as torch.full((n_layers,), config.rms_norm_eps, device=...).
Ensure n_layers matches kv.shape[1].
Use the materializer class __init__ which collects eps per layer automatically.

Example fix

// before
eps = torch.tensor(1e-5)
// after
eps = torch.full((n_layers,), 1e-5, device=kv.device)

Defensive patterns

Strategy: validation

Validate before calling

assert eps.shape == (n_layers,) and eps.numel() == kv.shape[1]

Prevention

Build eps with torch.full((n_layers,), cfg.rms_norm_eps).

When it happens

Trigger: Passing a scalar eps tensor, a Python-float, or an eps tensor whose length doesn't match the layer count in the stacked kv tensor.

Common situations: Building the fused inputs manually from model config where rms_norm_eps is a scalar and was not expanded per layer.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fbc7f55327653685. Report an issue: GitHub.