What causes "`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(out.shape)})." in sgl-project/sglang?

Allocate out = torch.empty(B, 1, HV, V, dtype=..., device=...) right before the call

sgl-project/sglang · error · ValueError

`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(

Error message

`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(out.shape)}).

What it means

The fused decode writes its output in place into a preallocated `out` buffer that must be exactly [B, 1, HV, V]. validate_packed_decode_inputs compares out.shape against the dims inferred from initial_state so the kernel's write addresses line up with the caller's buffer.

Source

Thrown at python/sglang/kernels/ops/attention/helion/kda_decode.py:308

            f"dimensions (got K={K}, V={V})."
        )
    if a.shape[1] != HV * K:
        raise ValueError(
            f"`a` must have shape [B, HV*K] with HV={HV}, K={K} "
            f"(got a.shape={tuple(a.shape)})."
        )
    if b.shape[1] != HV:
        raise ValueError(
            f"`b` must have shape [B, HV] with HV={HV} (got b.shape={tuple(b.shape)})."
        )
    if A_log.numel() != HV:
        raise ValueError(f"`A_log` must have {HV} elements (got {A_log.numel()}).")
    if dt_bias.numel() != HV * K:
        raise ValueError(
            f"`dt_bias` must have {HV * K} elements (got {dt_bias.numel()})."
        )
    if out.shape != (B, 1, HV, V):
        raise ValueError(
            f"`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(out.shape)})."
        )

    qkv_dim = mixed_qkv.shape[1]
    qk_dim = qkv_dim - HV * V
    if qk_dim <= 0 or qk_dim % 2 != 0:
        raise ValueError(
            f"Invalid packed `mixed_qkv` last dim={qkv_dim} for HV={HV}, V={V}."
        )
    q_dim = qk_dim // 2
    if q_dim % K != 0:
        raise ValueError(
            f"Invalid packed Q size {q_dim}: must be divisible by K={K}. "
            "KDA packed decode requires num_q_heads == num_k_heads and "
            "head_q_dim == head_k_dim."
        )
    H = q_dim // K
    if H <= 0 or HV % H != 0:

View on GitHub (pinned to 0132848349)

Solutions

Allocate out = torch.empty(B, 1, HV, V, dtype=..., device=...) right before the call
Check for swapped V/K dims — V is initial_state.shape[-2], K is shape[-1]
Never reuse a prefill-shaped buffer for the decode path

Example fix

// before
out = torch.empty(B, seqlen, HV, V, device=dev, dtype=dt)
// after
out = torch.empty(B, 1, HV, V, device=dev, dtype=dt)

Defensive patterns

Strategy: validation

Validate before calling

B, HV, V = mixed_qkv.size(0), initial_state.shape[-3], initial_state.shape[-2]
out = torch.empty(B, 1, HV, V, dtype=mixed_qkv.dtype, device=mixed_qkv.device)

Type guard

def valid_out(out: torch.Tensor, b: int, hv: int, v: int) -> bool:
    return out.shape == (b, 1, hv, v)

Prevention

Allocate `out` inside the decode step function, sized from current inputs
Never share output buffers across differing batch sizes or paths

When it happens

Trigger: Allocating out with an extra time dim (e.g. [B, T, HV, V] with T>1), transposed dims [B, 1, V, HV], or sized for a different batch than mixed_qkv.

Common situations: Adapting a prefill out buffer for decode; copying output allocation code from another backend with a different layout; cache reusing an out tensor allocated before batch size changed.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2606024ada7960c1. Report an issue: GitHub.