sgl-project/sglang · error · ValueError

The batch size is expected to be 1 rather than {q.shape[0]}

Error message

The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`.Please flatten variable-length inputs before processing.

What it means

chunk_gated_delta_rule supports variable-length sequences only via the cu_seqlens ragged format, which assumes all tokens are packed into a single batch row (batch size 1). If cu_seqlens is given and q.shape[0] != 1, inputs are not flattened and the kernel raises ValueError asking you to flatten them.

Source

Thrown at python/sglang/kernels/ops/attention/fla/chunk.py:234

    if head_first:
        raise DeprecationWarning(
            "head_first is deprecated and will be removed in a future version. "
            "Please use head_first=False for now instead."
        )
        q, k, v, beta, g = map(
            lambda x: rearrange(x, "b h t ... -> b t h ..."), (q, k, v, beta, g)
        )
    # if not head_first and q.shape[1] < q.shape[2]:
    #     warnings.warn(
    #         f"Input tensor shape suggests potential format mismatch: seq_len ({q.shape[1]}) < num_heads ({q.shape[2]}). "
    #         "This may indicate the inputs were passed in head-first format [B, H, T, ...] "
    #         "when head_first=False was specified. "
    #         "Please verify your input tensor format matches the expected shape [B, T, H, ...]."
    #     )
    if cu_seqlens is not None:
        if q.shape[0] != 1:
            raise ValueError(
                f"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`."
                f"Please flatten variable-length inputs before processing."
            )
        if (
            initial_state_indices is not None
            and initial_state_indices.shape[0] != len(cu_seqlens) - 1
        ):
            raise ValueError(
                f"The number of initial states is expected to be equal to the number of input sequences, "
                f"i.e., {len(cu_seqlens) - 1} rather than {initial_state_indices.shape[0]}."
            )
    if scale is None:
        scale = k.shape[-1] ** -0.5
    o, h = ChunkGatedDeltaRuleFunction.apply(
        q,
        k,
        v,
        g,

View on GitHub (pinned to 0132848349)

Solutions

1. Flatten q/k/v (and beta, g) to batch dim 1: q = q.reshape(1, -1, H) with tokens ordered to match cu_seqlens
2. Verify len(cu_seqlens) - 1 equals the number of packed sequences and initial_state_indices matches
3. Keep the padded-batch path and cu_seqlens mutually exclusive: use cu_seqlens only with flattened inputs

Example fix

# before
out = chunk_gated_delta_rule(q, k, v, beta, g, cu_seqlens=cu)  # q: [4, T, H]
# after
q = q.reshape(1, -1, q.shape[-1])  # same for k, v, beta, g, ordered per cu_seqlens
out = chunk_gated_delta_rule(q, k, v, beta, g, cu_seqlens=cu)

if cu_seqlens is not None:
    assert q.shape[0] == 1, "flatten variable-length inputs to [1, total_T, H] before using cu_seqlens"
    assert initial_state_indices is None or initial_state_indices.shape[0] == len(cu_seqlens) - 1

def varlen_ready(q: torch.Tensor, cu_seqlens) -> bool:
    return cu_seqlens is None or q.shape[0] == 1

When it happens

Trigger: Calling chunk_gated_delta_rule(q, ..., cu_seqlens=cu) where q is [B>1, T, H, ...] — i.e. passing a batched tensor together with cumulative-sequence-length indices.

Common situations: Migrating from per-sequence batching (padding) to varlen packing without reshaping; prefill paths feeding a [B, T, H] tensor while also computing cu_seqlens; forgetting q.reshape(1, -1, H) after concatenating sequences.

Related errors

• The number of initial states is expected to be equal to the
• head_first is deprecated and will be removed in a future ver
• The batch size is expected to be 1 rather than {q.shape[0]}
• {name} must be a 1D int32 or int64 tensor
• No frames were recorded

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5908a3f4d613ff6e. Report an issue: GitHub.