{"record":{"id":"4376b52e884e5ed5","repo":"sgl-project/sglang","slug":"the-batch-size-is-expected-to-be-1-rather-than-q-4376b5","errorCode":null,"errorMessage":"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`.Please flatten variable-length inputs before processing.","messagePattern":"The batch size is expected to be 1 rather than (.+?) when using `cu_seqlens`\\.Please flatten variable-length inputs before processing\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/fla/fused_recurrent.py","lineNumber":826,"sourceCode":"        >>> o, ht = fused_gated_recurrent_delta_rule(\n            q, k, v, g, beta,\n            initial_state=h0,\n            output_final_state=True\n        )\n        # for variable-length inputs, the batch size `B` is expected to be 1 and `cu_seqlens` is required\n        >>> q, k, v, g, beta = map(lambda x: rearrange(x, 'b t ... -> 1 (b t) ...'), (q, k, v, g, beta))\n        # for a batch with 4 sequences, `cu_seqlens` with 5 start/end positions are expected\n        >>> cu_seqlens = q.new_tensor([0, 2048, 4096, 6144, 8192], dtype=torch.long)\n        >>> o_var, ht_var = fused_gated_recurrent_delta_rule(\n            q, k, v, g, beta,\n            initial_state=h0,\n            output_final_state=True,\n            cu_seqlens=cu_seqlens\n        )\n    \"\"\"\n    if cu_seqlens is not None:\n        if q.shape[0] != 1:\n            raise ValueError(\n                f\"The batch size is expected to be 1 rather than {q.shape[0]} when using `cu_seqlens`.\"\n                f\"Please flatten variable-length inputs before processing.\"\n            )\n        if initial_state is not None and initial_state.shape[0] != len(cu_seqlens) - 1:\n            raise ValueError(\n                f\"The number of initial states is expected to be equal to the number of input sequences, \"\n                f\"i.e., {len(cu_seqlens) - 1} rather than {initial_state.shape[0]}.\"\n            )\n    if scale is None:\n        scale = k.shape[-1] ** -0.5\n    else:\n        assert scale > 0, \"scale must be positive\"\n    if beta is None:\n        beta = torch.ones_like(q[..., 0])\n    o, final_state = FusedRecurrentFunction.apply(\n        q,\n        k,\n        v,","sourceCodeStart":808,"sourceCodeEnd":844,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/fla/fused_recurrent.py#L808-L844","documentation":"When cu_seqlens (cumulative sequence lengths for variable-length packed inputs) is provided to fused_recurrent_gated_delta_rule, q/k/v must be flattened into a single [1, total_len, ...] batch. A leading batch dim other than 1 means the inputs are not flattened var-len format.","triggerScenarios":"Passing q with shape [B, T, ...] where B > 1 while also passing cu_seqlens, instead of concatenating all sequences into one row of length sum(cu_seqlens[1:]-cu_seqlens[:-1]).","commonSituations":"Migrating from chunked prefill code that used [B, T] padding; feeding ragged batches directly from a dataloader without flattening; mixing padded and var-len APIs.","solutions":["Flatten all variable-length sequences along the time dim and keep batch dim = 1: q = q.reshape(1, -1, D) with cu_seqlens marking boundaries","Or drop cu_seqlens and use padded [B, T] inputs if sequences are equal length","Ensure cu_seqlens is int32 on-device with length num_seqs+1 starting at 0"],"exampleFix":"// before\nq = torch.randn(B, T, D)  # B>1\nfused_recurrent_gated_delta_rule(q, k, v, cu_seqlens=cu)\n// after\nq = q.reshape(1, B*T, D)  # sequences concatenated in order\nfused_recurrent_gated_delta_rule(q, k, v, cu_seqlens=cu)","handlingStrategy":"validation","validationCode":"assert q.shape[0] == 1 or cu_seqlens is None\nif cu_seqlens is not None:\n    q = q.reshape(1, -1, q.shape[-1])","typeGuard":"def is_flattened_varlen(q: torch.Tensor, cu_seqlens) -> bool:\n    return cu_seqlens is None or q.shape[0] == 1","tryCatchPattern":null,"preventionTips":["Flatten ragged batches in the dataloader/collate, not at the kernel boundary","Standardize on [1, total_T, D] + cu_seqlens across the var-len code path"],"tags":["pytorch","variable-length","fla","linear-attention","validation"],"backgroundTag":"cu-seqlens-batch-not-flattened","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}