{"record":{"id":"ed5dddd5f800b8cc","repo":"sgl-project/sglang","slug":"flashinfer-cudnn-expects-packed-indptrs-as-a-torch","errorCode":null,"errorMessage":"flashinfer_cudnn expects packed indptrs as a torch.Tensor","messagePattern":"flashinfer_cudnn expects packed indptrs as a torch\\.Tensor","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/vision.py","lineNumber":700,"sourceCode":"            max_seqlen = kwargs[\"max_seqlen\"]\n\n        # max_seqlen must be python int\n        if isinstance(max_seqlen, torch.Tensor):\n            if max_seqlen.is_cuda:\n                max_seqlen = int(max_seqlen.detach().cpu().item())\n            else:\n                max_seqlen = int(max_seqlen.item())\n        else:\n            max_seqlen = int(max_seqlen)\n\n        # flatten if caller gives (b, s, h, d)\n        is_reshaped = q.dim() == 4\n        if is_reshaped:\n            reshape_batch_size = q.shape[0]\n            q, k, v = (rearrange(x, \"b s ... -> (b s) ...\") for x in [q, k, v])\n\n        if not isinstance(packed_cu_seqlens, torch.Tensor):\n            raise RuntimeError(\n                \"flashinfer_cudnn expects packed indptrs as a torch.Tensor\"\n            )\n\n        # sequence_lengths -> (B,)\n        if not isinstance(sequence_lengths, torch.Tensor):\n            raise RuntimeError(\"sequence_lengths must be a torch.Tensor\")\n        seq_lens_1d = sequence_lengths.view(-1).to(device=q.device, dtype=torch.int32)\n        B = int(seq_lens_1d.numel())\n\n        # cu_seqlens contains packed *element indptrs*:\n        # [qk_indptr(B+1), v_indptr(B+1), o_indptr(B+1)] => total 3*(B+1)\n        cu_seqlens_1d = packed_cu_seqlens.view(-1).to(\n            device=q.device, dtype=torch.int32\n        )\n        expected = 3 * (B + 1)\n        if int(cu_seqlens_1d.numel()) != expected:\n            raise RuntimeError(\n                f\"packed indptr numel mismatch: got {cu_seqlens_1d.numel()}, expected {expected} (= 3*(B+1))\"","sourceCodeStart":682,"sourceCodeEnd":718,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/vision.py#L682-L718","documentation":"The flashinfer_cudnn path feeds packed_cu_seqlens directly into the cudnn varlen attention API, which requires a torch.Tensor of int32 indptrs on the right device. If cu_seqlens arrives as a list, tuple, or numpy array (the format tolerated by other vision backends), it is rejected before the kernel launch to avoid a confusing low-level cudnn failure.","triggerScenarios":"Calling the flashinfer_cudnn vision forward with cu_seqlens that is not a torch.Tensor (e.g. the [gpu_tensor, seq_lens, max_seqlen] list used by the cuda-graph flash paths, or a Python list of lengths).","commonSituations":"Sharing argument-building code between flash3/flash4 (list format) and flashinfer_cudnn (tensor format) backends; passing numpy or list seqlens from a data preprocessing pipeline; backend swaps without adjusting the cu_seqlens format.","solutions":["Convert before the call: cu_seqlens = torch.tensor(..., dtype=torch.int32, device=q.device) or torch.as_tensor(list_form, ...).","Build backend-specific cu_seqlens in your wrapper rather than reusing one format across vision backends.","If cu_seqlens is the cuda-graph list, extract the tensor element ([0]) before passing to flashinfer_cudnn."],"exampleFix":"# before\nattn(q, k, v, cu_seqlens=[cu_seqlens_gpu, max_seqlen])\n# after\nattn(q, k, v, cu_seqlens=cu_seqlens_gpu_int32.to(q.device))","handlingStrategy":"type-guard","validationCode":"if not isinstance(cu_seqlens, torch.Tensor):\n    cu_seqlens = torch.as_tensor(\n        cu_seqlens[0] if isinstance(cu_seqlens, (list, tuple)) else cu_seqlens,\n        dtype=torch.int32,\n        device=q.device,\n    )","typeGuard":"def is_packed_indptr_tensor(x) -> bool:\n    return isinstance(x, torch.Tensor) and x.dtype == torch.int32 and x.is_cuda","tryCatchPattern":"try:\n    out = attn(q, k, v, cu_seqlens=cu_seqlens)\nexcept RuntimeError as e:\n    if \"packed indptrs\" in str(e):\n        cu = torch.as_tensor(cu_seqlens, dtype=torch.int32, device=q.device)\n        out = attn(q, k, v, cu_seqlens=cu)\n    else:\n        raise","preventionTips":["Normalize cu_seqlens to an int32 CUDA tensor immediately after preprocessing.","Never forward the cuda-graph list format into flashinfer_cudnn.","Add dtype/device assertions on indptrs before attention calls in tests."],"tags":["sglang","vision-transformer","flashinfer-cudnn","type-validation","cu-seqlens"],"backgroundTag":"wrong-argument-type","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}