sgl-project/sglang · error · ValueError

Invalid v_out shape for fused KV materialization: got

Error message

Invalid v_out shape for fused KV materialization: got {tuple(v_out.shape)}, expected {expected_shape}.

What it means

Output-buffer contract for fused KV materialization: v_out's shape must equal the expected (tokens, heads, head_dim) layout. A mis-shaped v_out indicates the caller sized it for a different layer count or token budget.

Solutions

1. Allocate v_out identically to k_out (torch.empty_like(k_out)) or pass None.
2. Verify total_ctx and n_layers match the kv input.

Example fix

// before
v_out = torch.empty(total_ctx, H, D)
// after
v_out = None  # or k_out.new_empty((n_layers, total_ctx, H, D))

expected = (n_layers, total_ctx, num_kv_heads, head_dim)
assert v_out is None or v_out.shape == expected

When it happens

Trigger: Passing v_out sized or ordered differently from k_out (e.g. only [total_ctx, ...] or sized for a shorter context).

Common situations: Reusing stale cache buffers sized for a previous batch, or mirroring an incorrect k_out allocation into v_out.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

• D= must be divisible by GROUP_SIZE=
• head_dim mismatch: unified_kv=
• Invalid k_out shape for fused KV materialization: got
• k_pool has incompatible shape
• kv-canary: RealKvSource.tensor must be at least 2-D, got…

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/555b395489c9c097. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:201

    if k_out is None:
        k_out = torch.empty(expected_shape, dtype=kv.dtype, device=kv.device)
    else:
        if k_out.shape != expected_shape:
            raise ValueError(
                "Invalid k_out shape for fused KV materialization: "
                f"got {tuple(k_out.shape)}, expected {expected_shape}."
            )
        if k_out.device != kv.device or k_out.dtype != kv.dtype:
            raise ValueError(
                "Invalid k_out device/dtype for fused KV materialization: "
                f"got device={k_out.device}, dtype={k_out.dtype}, "
                f"expected device={kv.device}, dtype={kv.dtype}."
            )
    if v_out is None:
        v_out = torch.empty_like(k_out)
    else:
        if v_out.shape != expected_shape:
            raise ValueError(
                "Invalid v_out shape for fused KV materialization: "
                f"got {tuple(v_out.shape)}, expected {expected_shape}."
            )
        if v_out.device != kv.device or v_out.dtype != kv.dtype:
            raise ValueError(
                "Invalid v_out device/dtype for fused KV materialization: "
                f"got device={v_out.device}, dtype={v_out.dtype}, "
                f"expected device={kv.device}, dtype={kv.dtype}."
            )

    _fused_norm_rope_kernel_stacked[(total_ctx, num_kv_heads, n_layers)](
        kv,
        k_norm_weight,
        eps,
        cos_sin_cache,
        positions,
        k_out,
        v_out,

View on GitHub (pinned to 0132848349)