sgl-project/sglang · error · ValueError

The number of intermediate state indices is expected to be…

Error message

The number of intermediate state indices is expected to be equal to the number of input sequences, i.e., {initial_state_indices.shape[0]} != {intermediate_state_indices.shape[0]}.

What it means

When both initial_state_indices and intermediate_state_indices are passed to fused_recurrent_gated_delta_rule_update with cu_seqlens, they must have identical lengths (one per sequence). This check fires when the intermediate (output) state slot tensor doesn't line up with the input state slots.

Solutions

  1. Allocate both index tensors with shape [len(cu_seqlens)-1]
  2. When flushing per-token states, group them per sequence or use the appropriate API variant
  3. Log both shapes before the call during development

Example fix

// before
initial_state_indices = torch.arange(N)  # [N]
intermediate_state_indices = torch.arange(N*T)  # [N*T] -> mismatch
// after
initial_state_indices = torch.arange(N)
intermediate_state_indices = flush_slots[:N]  # [N], one per sequence
Defensive patterns

Strategy: validation

Validate before calling

assert initial_state_indices.shape == intermediate_state_indices.shape, 'slot tensors must match'

Type guard

def valid_slot_pair(init_idx, inter_idx) -> bool:
    return init_idx.ndim == 1 and init_idx.shape == inter_idx.shape

Prevention

When it happens

Trigger: Allocating the output/intermediate state index tensor with a different batch length than the input index tensor, e.g. one per token instead of per sequence.

Common situations: Reworking a state cache: input slots sized per-sequence but flush slots sized per-token or per-head; partial updates after sequences finish mid-batch.

Related errors


AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/692f6e9647251b83. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:1232

    intermediate_states_buffer: Optional[torch.Tensor] = None,
    intermediate_state_indices: Optional[torch.Tensor] = None,
    cache_steps: Optional[int] = None,
    retrieve_parent_token: Optional[torch.Tensor] = None,
) -> torch.Tensor:
    if cu_seqlens is not None:
        if q.shape[0] != 1:
            raise ValueError(
                f"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`."
                f"Please flatten variable-length inputs before processing."
            )
        if initial_state_source is not None:
            if initial_state_indices.shape[0] != len(cu_seqlens) - 1:
                raise ValueError(
                    f"The number of initial states is expected to be equal to the number of input sequences, "
                    f"i.e., {len(cu_seqlens) - 1} rather than {initial_state_indices.shape[0]}."
                )
            if initial_state_indices.shape[0] != intermediate_state_indices.shape[0]:
                raise ValueError(
                    f"The number of intermediate state indices is expected to be equal to the number of input sequences, "
                    f"i.e., {initial_state_indices.shape[0]} != {intermediate_state_indices.shape[0]}."
                )
    if scale is None:
        scale = k.shape[-1] ** -0.5
    else:
        assert scale > 0, "scale must be positive"
    if beta is None:
        beta = torch.ones_like(q[..., 0])
    o = FusedRecurrentUpdateFunction.apply(
        q,
        k,
        v,
        g,
        beta,
        scale,
        initial_state_source,
        initial_state_indices,

View on GitHub (pinned to 0132848349)