sgl-project/sglang · error · ValueError

`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(

Error message

`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(out.shape)}).

What it means

The caller-preallocated output must have exactly shape (B, 1, HV, V) — one decode step per token, HV value heads, and head_dim V — where B, HV, V are all derived from the other validated inputs. The wrapper raises when out has any other shape, such as the flattened (B, HV*V) or head-first (B, HV, 1, V) layouts.

Source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:337

        )

    if initial_state.ndim != 4:
        raise ValueError(
            f"`initial_state` must be a 4D tensor (got ndim={initial_state.ndim})."
        )
    if initial_state.stride(-1) != 1:
        raise ValueError("`initial_state` must be contiguous in the last dim.")
    HV, V, K = initial_state.shape[-3:]
    if a.shape[1] != HV or b.shape[1] != HV:
        raise ValueError(
            f"`a`/`b` must have shape [B, HV] with HV={HV} (got a.shape={tuple(a.shape)}, b.shape={tuple(b.shape)})."
        )
    if A_log.numel() != HV or dt_bias.numel() != HV:
        raise ValueError(
            f"`A_log` and `dt_bias` must have {HV} elements (got A_log.numel()={A_log.numel()}, dt_bias.numel()={dt_bias.numel()})."
        )
    if out.shape != (B, 1, HV, V):
        raise ValueError(
            f"`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(out.shape)})."
        )

    qkv_dim = mixed_qkv.shape[1]
    qk_dim = qkv_dim - HV * V
    if qk_dim <= 0 or qk_dim % 2 != 0:
        raise ValueError(
            f"Invalid packed `mixed_qkv` last dim={qkv_dim} for HV={HV}, V={V}."
        )
    q_dim = qk_dim // 2
    if q_dim % K != 0:
        raise ValueError(f"Invalid packed Q size {q_dim}: must be divisible by K={K}.")
    H = q_dim // K
    if H <= 0 or HV % H != 0:
        raise ValueError(
            f"Invalid head config inferred from mixed_qkv: H={H}, HV={HV}."
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate out = torch.empty((B, 1, HV, V), dtype=qkv.dtype, device=dev) with HV/V taken from initial_state.shape[-3]/shape[-2]
2. Flatten afterwards if a 2D result is needed: out.view(B, HV*V)

Example fix

# before
out = torch.empty(B, HV*V, device=dev, dtype=dt)
# after
out = torch.empty(B, 1, HV, V, device=dev, dtype=dt)
out, _ = fused_recurrent_gated_delta_rule_packed_decode(..., out=out, ...)
hidden = out.view(B, HV*V)

B, HV, V = mixed_qkv.shape[0], initial_state.shape[-3], initial_state.shape[-2]
expected = (B, 1, HV, V)
assert out.shape == expected, (out.shape, expected)

def out_shape_ok(out, mixed_qkv, initial_state) -> bool:
    return out.shape == (mixed_qkv.shape[0], 1, initial_state.shape[-3], initial_state.shape[-2])

When it happens

Trigger: Allocating out as (B, HV*V) to skip a later reshape; reusing a (B, T, HV, V) prefill-shaped buffer; using V and K swapped relative to the state tensor's (HV, V, K).

Common situations: Custom decode loops optimizing away reshapes; buffers created before a config change of num_v_heads/head_dim; porting from fla's (B, H, T, D) output convention.

Related errors

• `mixed_qkv` must be a 2D tensor (got ndim={mixed_qkv.ndim}).
• `a` and `b` must be 2D tensors (got a.ndim={a.ndim}, b.ndim=
• `A_log`/`dt_bias` must be 1D tensors.
• `ssm_state_indices` must be 1D for packed decode (got ndim={
• `out` must be contiguous.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9763c1308463a64a. Report an issue: GitHub.