sgl-project/sglang · error · ValueError
The number of intermediate state indices is expected to be…
Error message
The number of intermediate state indices is expected to be equal to the number of input sequences, i.e., {initial_state_indices.shape[0]} != {intermediate_state_indices.shape[0]}. What it means
When both initial_state_indices and intermediate_state_indices are passed to fused_recurrent_gated_delta_rule_update with cu_seqlens, they must have identical lengths (one per sequence). This check fires when the intermediate (output) state slot tensor doesn't line up with the input state slots.
Solutions
- Allocate both index tensors with shape [len(cu_seqlens)-1]
- When flushing per-token states, group them per sequence or use the appropriate API variant
- Log both shapes before the call during development
Example fix
// before initial_state_indices = torch.arange(N) # [N] intermediate_state_indices = torch.arange(N*T) # [N*T] -> mismatch // after initial_state_indices = torch.arange(N) intermediate_state_indices = flush_slots[:N] # [N], one per sequence
Defensive patterns
Strategy: validation
Validate before calling
assert initial_state_indices.shape == intermediate_state_indices.shape, 'slot tensors must match'
Type guard
def valid_slot_pair(init_idx, inter_idx) -> bool:
return init_idx.ndim == 1 and init_idx.shape == inter_idx.shape Prevention
- Allocate input and flush slot tensors together from one shape
- Add a debug assert in the serving loop
When it happens
Trigger: Allocating the output/intermediate state index tensor with a different batch length than the input index tensor, e.g. one per token instead of per sequence.
Common situations: Reworking a state cache: input slots sized per-sequence but flush slots sized per-token or per-head; partial updates after sequences finish mid-batch.
Related errors
- The number of initial states is expected to be equal to the…
- `dt_bias` must have elements (got ).
- `initial_state` must be 4D
- `mixed_qkv` must be 2D
- pairs length must be greater than 0
AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28).
Data as JSON: /api/errors/692f6e9647251b83.
Report an issue: GitHub.
Appendix: source
Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:1232
intermediate_states_buffer: Optional[torch.Tensor] = None,
intermediate_state_indices: Optional[torch.Tensor] = None,
cache_steps: Optional[int] = None,
retrieve_parent_token: Optional[torch.Tensor] = None,
) -> torch.Tensor:
if cu_seqlens is not None:
if q.shape[0] != 1:
raise ValueError(
f"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`."
f"Please flatten variable-length inputs before processing."
)
if initial_state_source is not None:
if initial_state_indices.shape[0] != len(cu_seqlens) - 1:
raise ValueError(
f"The number of initial states is expected to be equal to the number of input sequences, "
f"i.e., {len(cu_seqlens) - 1} rather than {initial_state_indices.shape[0]}."
)
if initial_state_indices.shape[0] != intermediate_state_indices.shape[0]:
raise ValueError(
f"The number of intermediate state indices is expected to be equal to the number of input sequences, "
f"i.e., {initial_state_indices.shape[0]} != {intermediate_state_indices.shape[0]}."
)
if scale is None:
scale = k.shape[-1] ** -0.5
else:
assert scale > 0, "scale must be positive"
if beta is None:
beta = torch.ones_like(q[..., 0])
o = FusedRecurrentUpdateFunction.apply(
q,
k,
v,
g,
beta,
scale,
initial_state_source,
initial_state_indices,View on GitHub (pinned to 0132848349)