sgl-project/sglang · error · ValueError

`initial_state` must be contiguous in the last dim.

Error message

`initial_state` must be contiguous in the last dim.

What it means

Like the other operand checks, the recurrent state tensor must have unit stride in its last dim (K) so the kernel can vectorize loads/stores of state rows. The wrapper raises when initial_state.stride(-1) != 1, e.g. a state cache stored transposed or a strided slice over the K dimension.

Source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:326

        raise ValueError("All inputs must be on the same device.")

    B = mixed_qkv.shape[0]
    if a.shape[0] != B or b.shape[0] != B:
        raise ValueError(
            "Mismatched batch sizes: "
            f"mixed_qkv.shape[0]={B}, a.shape[0]={a.shape[0]}, b.shape[0]={b.shape[0]}."
        )
    if ssm_state_indices.shape[0] != B:
        raise ValueError(
            f"`ssm_state_indices` must have shape [B] (got {tuple(ssm_state_indices.shape)}; expected ({B},))."
        )

    if initial_state.ndim != 4:
        raise ValueError(
            f"`initial_state` must be a 4D tensor (got ndim={initial_state.ndim})."
        )
    if initial_state.stride(-1) != 1:
        raise ValueError("`initial_state` must be contiguous in the last dim.")
    HV, V, K = initial_state.shape[-3:]
    if a.shape[1] != HV or b.shape[1] != HV:
        raise ValueError(
            f"`a`/`b` must have shape [B, HV] with HV={HV} (got a.shape={tuple(a.shape)}, b.shape={tuple(b.shape)})."
        )
    if A_log.numel() != HV or dt_bias.numel() != HV:
        raise ValueError(
            f"`A_log` and `dt_bias` must have {HV} elements (got A_log.numel()={A_log.numel()}, dt_bias.numel()={dt_bias.numel()})."
        )
    if out.shape != (B, 1, HV, V):
        raise ValueError(
            f"`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(out.shape)})."
        )

    qkv_dim = mixed_qkv.shape[1]
    qk_dim = qkv_dim - HV * V
    if qk_dim <= 0 or qk_dim % 2 != 0:
        raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Transpose to (N, HV, V, K) with .permute(...) followed by .contiguous() once at cache-allocation time
2. Keep the canonical layout when allocating the cache: torch.empty(num_states, HV, V, K)

Example fix

# before
state = cache[layer]  # (N, HV, K, V) stored transposed
# after
state = cache[layer].permute(0, 1, 3, 2).contiguous()  # (N, HV, V, K)

if initial_state.stride(-1) != 1:
    initial_state = initial_state.contiguous()

def state_last_contig(s: torch.Tensor) -> bool:
    return s.ndim == 4 and s.stride(-1) == 1

When it happens

Trigger: State caches laid out (num_states, K, V, HV) passed without transpose; slicing a bigger state buffer over the last dim; states materialized from flattened views with non-compact K.

Common situations: Custom mamba cache formats migrated from other frameworks (e.g. (K,V) transposed layouts); TP sharding that slices the K dim producing strided rows.

Related errors

• `mixed_qkv` must be contiguous in the last dim.
• `a`/`b` must be contiguous in the last dim.
• `A_log`/`dt_bias` must be contiguous.
• `ssm_state_indices` must be 1D for packed decode (got ndim={
• `out` must be contiguous.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/806b7bc6d1a8b594. Report an issue: GitHub.