{"record":{"id":"0611056615dfd904","repo":"sgl-project/sglang","slug":"cuda-graph-mode-cu-seqlens-should-be-a-list","errorCode":null,"errorMessage":"cuda-graph mode cu_seqlens should be a list","messagePattern":"cuda-graph mode cu_seqlens should be a list","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/vision.py","lineNumber":486,"sourceCode":"        **kwargs,\n    ) -> torch.Tensor:\n        r\"\"\"\n        Args:\n            cu_seqlens: [b]\n        Returns:\n             [b * s, h, head_size]\n        \"\"\"\n        if forward_metadata is not None:\n            cu_seqlens_gpu = forward_metadata.cu_seqlens\n            seq_lens = forward_metadata.seq_lens\n            max_seqlen = forward_metadata.max_seqlen\n            output = torch.empty_like(q)\n        elif envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get():\n            if \"output_ws\" not in kwargs:\n                raise RuntimeError(\"output_ws should be prepared for cuda-graph mode\")\n\n            if not isinstance(cu_seqlens, list):\n                raise RuntimeError(\"cuda-graph mode cu_seqlens should be a list\")\n\n            output = kwargs[\"output_ws\"]\n            cu_seqlens_gpu = cu_seqlens[0]\n            seq_lens = cu_seqlens[1]\n            max_seqlen = cu_seqlens[2]\n        else:\n            cu_seqlens_gpu = resolve_seqlens(cu_seqlens, bsz, seq_len, device=q.device)\n            seq_lens = kwargs.get(\"sequence_lengths\")\n            if seq_lens is None:\n                seq_lens = cu_seqlens_gpu[1:] - cu_seqlens_gpu[:-1]\n            else:\n                seq_lens = seq_lens.to(device=q.device, dtype=torch.int32)\n            max_seqlen = resolve_precomputed_max_seqlen(\n                cu_seqlens_gpu, kwargs.get(\"max_seqlen\")\n            )\n            # [b * s, head, head_size]\n            output = torch.empty_like(q)\n","sourceCodeStart":468,"sourceCodeEnd":504,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/vision.py#L468-L504","documentation":"In cuda-graph mode the flash3 ViT forward expects cu_seqlens to be a list/tuple bundling [cu_seqlens_gpu, seq_lens, max_seqlen] — the three graph-stable tensors captured once at graph capture time. Passing a bare tensor (the non-cuda-graph format) is rejected because the code needs to index positions 0/1/2.","triggerScenarios":"SGLANG_VIT_ENABLE_CUDA_GRAPH enabled and forward called with cu_seqlens as a torch.Tensor or other non-list (e.g. the plain cu_seqlens tensor used on the normal path) and forward_metadata None.","commonSituations":"Reusing a non-graph capture path's argument format when cuda-graph was enabled; custom vision-model integrations that pass only a cu_seqlens tensor; env var enabled globally while a model only supports the legacy call signature.","solutions":["Pass a 3-element list: cu_seqlens=[cu_seqlens_gpu_tensor, seq_lens_tensor, max_seqlen_value] when cuda-graph mode is on.","Mirror how SGLang's own runner builds the argument during graph capture (bundle the three prepared tensors).","Disable SGLANG_VIT_ENABLE_CUDA_GRAPH if your caller cannot provide the bundled format."],"exampleFix":"# before\nattn(q, k, v, cu_seqlens=cu_seqlens_tensor)\n# after\nattn(q, k, v, cu_seqlens=[cu_seqlens_gpu, seq_lens, max_seqlen])","handlingStrategy":"type-guard","validationCode":"if envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get() and not isinstance(cu_seqlens, (list, tuple)) or len(cu_seqlens) != 3:\n    cu_seqlens = [cu_seqlens_gpu, seq_lens, max_seqlen]  # bundle graph-stable tensors","typeGuard":"def is_cg_cu_seqlens_flash3(x) -> bool:\n    return isinstance(x, (list, tuple)) and len(x) == 3 and isinstance(x[0], torch.Tensor)","tryCatchPattern":null,"preventionTips":["Keep one function per backend that assembles its expected cu_seqlens format.","Assert the format right before the forward call in debug builds.","Document which env vars change call signatures for your vision model."],"tags":["sglang","vision-transformer","cuda-graph","type-validation","cu-seqlens"],"backgroundTag":"wrong-argument-type","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}