sgl-project/sglang · error · RuntimeError

flashinfer_cudnn expects packed indptrs as a torch.Tensor

Error message

flashinfer_cudnn expects packed indptrs as a torch.Tensor

What it means

The flashinfer_cudnn path feeds packed_cu_seqlens directly into the cudnn varlen attention API, which requires a torch.Tensor of int32 indptrs on the right device. If cu_seqlens arrives as a list, tuple, or numpy array (the format tolerated by other vision backends), it is rejected before the kernel launch to avoid a confusing low-level cudnn failure.

Source

Thrown at python/sglang/srt/layers/attention/vision.py:700

            max_seqlen = kwargs["max_seqlen"]

        # max_seqlen must be python int
        if isinstance(max_seqlen, torch.Tensor):
            if max_seqlen.is_cuda:
                max_seqlen = int(max_seqlen.detach().cpu().item())
            else:
                max_seqlen = int(max_seqlen.item())
        else:
            max_seqlen = int(max_seqlen)

        # flatten if caller gives (b, s, h, d)
        is_reshaped = q.dim() == 4
        if is_reshaped:
            reshape_batch_size = q.shape[0]
            q, k, v = (rearrange(x, "b s ... -> (b s) ...") for x in [q, k, v])

        if not isinstance(packed_cu_seqlens, torch.Tensor):
            raise RuntimeError(
                "flashinfer_cudnn expects packed indptrs as a torch.Tensor"
            )

        # sequence_lengths -> (B,)
        if not isinstance(sequence_lengths, torch.Tensor):
            raise RuntimeError("sequence_lengths must be a torch.Tensor")
        seq_lens_1d = sequence_lengths.view(-1).to(device=q.device, dtype=torch.int32)
        B = int(seq_lens_1d.numel())

        # cu_seqlens contains packed *element indptrs*:
        # [qk_indptr(B+1), v_indptr(B+1), o_indptr(B+1)] => total 3*(B+1)
        cu_seqlens_1d = packed_cu_seqlens.view(-1).to(
            device=q.device, dtype=torch.int32
        )
        expected = 3 * (B + 1)
        if int(cu_seqlens_1d.numel()) != expected:
            raise RuntimeError(
                f"packed indptr numel mismatch: got {cu_seqlens_1d.numel()}, expected {expected} (= 3*(B+1))"

View on GitHub (pinned to 0132848349)

Solutions

1. Convert before the call: cu_seqlens = torch.tensor(..., dtype=torch.int32, device=q.device) or torch.as_tensor(list_form, ...).
2. Build backend-specific cu_seqlens in your wrapper rather than reusing one format across vision backends.
3. If cu_seqlens is the cuda-graph list, extract the tensor element ([0]) before passing to flashinfer_cudnn.

Example fix

# before
attn(q, k, v, cu_seqlens=[cu_seqlens_gpu, max_seqlen])
# after
attn(q, k, v, cu_seqlens=cu_seqlens_gpu_int32.to(q.device))

if not isinstance(cu_seqlens, torch.Tensor):
    cu_seqlens = torch.as_tensor(
        cu_seqlens[0] if isinstance(cu_seqlens, (list, tuple)) else cu_seqlens,
        dtype=torch.int32,
        device=q.device,
    )

def is_packed_indptr_tensor(x) -> bool:
    return isinstance(x, torch.Tensor) and x.dtype == torch.int32 and x.is_cuda

try:
    out = attn(q, k, v, cu_seqlens=cu_seqlens)
except RuntimeError as e:
    if "packed indptrs" in str(e):
        cu = torch.as_tensor(cu_seqlens, dtype=torch.int32, device=q.device)
        out = attn(q, k, v, cu_seqlens=cu)
    else:
        raise

When it happens

Trigger: Calling the flashinfer_cudnn vision forward with cu_seqlens that is not a torch.Tensor (e.g. the [gpu_tensor, seq_lens, max_seqlen] list used by the cuda-graph flash paths, or a Python list of lengths).

Common situations: Sharing argument-building code between flash3/flash4 (list format) and flashinfer_cudnn (tensor format) backends; passing numpy or list seqlens from a data preprocessing pipeline; backend swaps without adjusting the cu_seqlens format.

Understand the failure class

Background: "Wrong argument type", "must be a string", "expected Array or Prism::Scope": TypeError and ArgumentError when a library receives a value of the wrong type — this error's family across 28 libraries.

Related errors

• cuda-graph mode cu_seqlens should be a list
• sequence_lengths should be prepared for vision flashinfer_cu
• max_seqlen should be prepared for vision flashinfer_cudnn at
• actions must be a list[list[str]]
• Inkling reasoning_effort must not be a boolean

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ed5dddd5f800b8cc. Report an issue: GitHub.