sgl-project/sglang · error · ValueError

The number of initial states is expected to be equal to the

Error message

The number of initial states is expected to be equal to the number of input sequences, i.e., {len(cu_seqlens) - 1} rather than {initial_state_indices.shape[0]}.

What it means

In fused_recurrent_gated_delta_rule_update, when initial_state_source is provided with cu_seqlens, initial_state_indices must have exactly one row per input sequence, i.e. len(cu_seqlens)-1 entries. The mismatch means the state-index tensor doesn't cover every sequence in the packed batch.

Source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:1227

    initial_state_indices: torch.Tensor = None,
    cu_seqlens: Optional[torch.LongTensor] = None,
    use_qk_l2norm_in_kernel: bool = False,
    disable_state_update: bool = False,
    disable_output_calculation: bool = False,
    intermediate_states_buffer: Optional[torch.Tensor] = None,
    intermediate_state_indices: Optional[torch.Tensor] = None,
    cache_steps: Optional[int] = None,
    retrieve_parent_token: Optional[torch.Tensor] = None,
) -> torch.Tensor:
    if cu_seqlens is not None:
        if q.shape[0] != 1:
            raise ValueError(
                f"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`."
                f"Please flatten variable-length inputs before processing."
            )
        if initial_state_source is not None:
            if initial_state_indices.shape[0] != len(cu_seqlens) - 1:
                raise ValueError(
                    f"The number of initial states is expected to be equal to the number of input sequences, "
                    f"i.e., {len(cu_seqlens) - 1} rather than {initial_state_indices.shape[0]}."
                )
            if initial_state_indices.shape[0] != intermediate_state_indices.shape[0]:
                raise ValueError(
                    f"The number of intermediate state indices is expected to be equal to the number of input sequences, "
                    f"i.e., {initial_state_indices.shape[0]} != {intermediate_state_indices.shape[0]}."
                )
    if scale is None:
        scale = k.shape[-1] ** -0.5
    else:
        assert scale > 0, "scale must be positive"
    if beta is None:
        beta = torch.ones_like(q[..., 0])
    o = FusedRecurrentUpdateFunction.apply(
        q,
        k,
        v,

View on GitHub (pinned to 0132848349)

Solutions

1. Resize/refresh initial_state_indices to length len(cu_seqlens)-1 each step
2. Verify cu_seqlens has num_seqs+1 entries starting at 0 and ending at total_T
3. Index into the state pool per sequence before the call

Example fix

// before
initial_state_indices = pool_idx  # stale length from previous batch
// after
initial_state_indices = torch.arange(num_seqs, dtype=torch.int32, device=dev)  # len == len(cu_seqlens)-1

n = len(cu_seqlens) - 1
assert initial_state_indices.shape[0] == n
assert initial_state_source.shape[0] >= n

When it happens

Trigger: Passing initial_state_indices with batch size different from the number of var-len segments (e.g. reusing a per-request buffer sized from a previous batch).

Common situations: Serving with dynamic batch sizes where the index tensor is a persistent pool buffer not resized per step; off-by-one when building cu_seqlens.

Related errors

• The number of intermediate state indices is expected to be e
• The batch size is expected to be 1 rather than {q.shape[0]}
• `dt_bias` must have {HV * K} elements (got {dt_bias.numel()}
• `mixed_qkv` must be 2D (got ndim={mixed_qkv.ndim}).
• `initial_state` must be 4D (got ndim={initial_state.ndim}).

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eb65ae4ab1115f33. Report an issue: GitHub.