sgl-project/sglang · error · ValueError

The number of intermediate state indices is expected to be e

Error message

The number of intermediate state indices is expected to be equal to the number of input sequences, i.e., {initial_state_indices.shape[0]} != {intermediate_state_indices.shape[0]}.

What it means

When both initial_state_indices and intermediate_state_indices are passed to fused_recurrent_gated_delta_rule_update with cu_seqlens, they must have identical lengths (one per sequence). This check fires when the intermediate (output) state slot tensor doesn't line up with the input state slots.

Source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:1232

    intermediate_states_buffer: Optional[torch.Tensor] = None,
    intermediate_state_indices: Optional[torch.Tensor] = None,
    cache_steps: Optional[int] = None,
    retrieve_parent_token: Optional[torch.Tensor] = None,
) -> torch.Tensor:
    if cu_seqlens is not None:
        if q.shape[0] != 1:
            raise ValueError(
                f"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`."
                f"Please flatten variable-length inputs before processing."
            )
        if initial_state_source is not None:
            if initial_state_indices.shape[0] != len(cu_seqlens) - 1:
                raise ValueError(
                    f"The number of initial states is expected to be equal to the number of input sequences, "
                    f"i.e., {len(cu_seqlens) - 1} rather than {initial_state_indices.shape[0]}."
                )
            if initial_state_indices.shape[0] != intermediate_state_indices.shape[0]:
                raise ValueError(
                    f"The number of intermediate state indices is expected to be equal to the number of input sequences, "
                    f"i.e., {initial_state_indices.shape[0]} != {intermediate_state_indices.shape[0]}."
                )
    if scale is None:
        scale = k.shape[-1] ** -0.5
    else:
        assert scale > 0, "scale must be positive"
    if beta is None:
        beta = torch.ones_like(q[..., 0])
    o = FusedRecurrentUpdateFunction.apply(
        q,
        k,
        v,
        g,
        beta,
        scale,
        initial_state_source,
        initial_state_indices,

View on GitHub (pinned to 0132848349)

Solutions

Allocate both index tensors with shape [len(cu_seqlens)-1]
When flushing per-token states, group them per sequence or use the appropriate API variant
Log both shapes before the call during development

Example fix

// before
initial_state_indices = torch.arange(N)  # [N]
intermediate_state_indices = torch.arange(N*T)  # [N*T] -> mismatch
// after
initial_state_indices = torch.arange(N)
intermediate_state_indices = flush_slots[:N]  # [N], one per sequence

Defensive patterns

Strategy: validation

Validate before calling

assert initial_state_indices.shape == intermediate_state_indices.shape, 'slot tensors must match'

Type guard

def valid_slot_pair(init_idx, inter_idx) -> bool:
    return init_idx.ndim == 1 and init_idx.shape == inter_idx.shape

Prevention

Allocate input and flush slot tensors together from one shape
Add a debug assert in the serving loop

When it happens

Trigger: Allocating the output/intermediate state index tensor with a different batch length than the input index tensor, e.g. one per token instead of per sequence.

Common situations: Reworking a state cache: input slots sized per-sequence but flush slots sized per-token or per-head; partial updates after sequences finish mid-batch.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/692f6e9647251b83. Report an issue: GitHub.