sgl-project/sglang · error · ValueError

The batch size is expected to be 1 rather than {q.shape[0]}

Error message

The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`.Please flatten variable-length inputs before processing.

What it means

When cu_seqlens (cumulative sequence lengths for variable-length packed inputs) is provided to fused_recurrent_gated_delta_rule, q/k/v must be flattened into a single [1, total_len, ...] batch. A leading batch dim other than 1 means the inputs are not flattened var-len format.

Source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:826

        >>> o, ht = fused_gated_recurrent_delta_rule(
            q, k, v, g, beta,
            initial_state=h0,
            output_final_state=True
        )
        # for variable-length inputs, the batch size `B` is expected to be 1 and `cu_seqlens` is required
        >>> q, k, v, g, beta = map(lambda x: rearrange(x, 'b t ... -> 1 (b t) ...'), (q, k, v, g, beta))
        # for a batch with 4 sequences, `cu_seqlens` with 5 start/end positions are expected
        >>> cu_seqlens = q.new_tensor([0, 2048, 4096, 6144, 8192], dtype=torch.long)
        >>> o_var, ht_var = fused_gated_recurrent_delta_rule(
            q, k, v, g, beta,
            initial_state=h0,
            output_final_state=True,
            cu_seqlens=cu_seqlens
        )
    """
    if cu_seqlens is not None:
        if q.shape[0] != 1:
            raise ValueError(
                f"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`."
                f"Please flatten variable-length inputs before processing."
            )
        if initial_state is not None and initial_state.shape[0] != len(cu_seqlens) - 1:
            raise ValueError(
                f"The number of initial states is expected to be equal to the number of input sequences, "
                f"i.e., {len(cu_seqlens) - 1} rather than {initial_state.shape[0]}."
            )
    if scale is None:
        scale = k.shape[-1] ** -0.5
    else:
        assert scale > 0, "scale must be positive"
    if beta is None:
        beta = torch.ones_like(q[..., 0])
    o, final_state = FusedRecurrentFunction.apply(
        q,
        k,
        v,

View on GitHub (pinned to 0132848349)

Solutions

Flatten all variable-length sequences along the time dim and keep batch dim = 1: q = q.reshape(1, -1, D) with cu_seqlens marking boundaries
Or drop cu_seqlens and use padded [B, T] inputs if sequences are equal length
Ensure cu_seqlens is int32 on-device with length num_seqs+1 starting at 0

Example fix

// before
q = torch.randn(B, T, D)  # B>1
fused_recurrent_gated_delta_rule(q, k, v, cu_seqlens=cu)
// after
q = q.reshape(1, B*T, D)  # sequences concatenated in order
fused_recurrent_gated_delta_rule(q, k, v, cu_seqlens=cu)

Defensive patterns

Strategy: validation

Validate before calling

assert q.shape[0] == 1 or cu_seqlens is None
if cu_seqlens is not None:
    q = q.reshape(1, -1, q.shape[-1])

Type guard

def is_flattened_varlen(q: torch.Tensor, cu_seqlens) -> bool:
    return cu_seqlens is None or q.shape[0] == 1

Prevention

Flatten ragged batches in the dataloader/collate, not at the kernel boundary
Standardize on [1, total_T, D] + cu_seqlens across the var-len code path

When it happens

Trigger: Passing q with shape [B, T, ...] where B > 1 while also passing cu_seqlens, instead of concatenating all sequences into one row of length sum(cu_seqlens[1:]-cu_seqlens[:-1]).

Common situations: Migrating from chunked prefill code that used [B, T] padding; feeding ragged batches directly from a dataloader without flattening; mixing padded and var-len APIs.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4376b52e884e5ed5. Report an issue: GitHub.