sgl-project/sglang · error · RuntimeError

cuda-graph mode cu_seqlens should be a list

Error message

cuda-graph mode cu_seqlens should be a list

What it means

In cuda-graph mode the flash3 ViT forward expects cu_seqlens to be a list/tuple bundling [cu_seqlens_gpu, seq_lens, max_seqlen] — the three graph-stable tensors captured once at graph capture time. Passing a bare tensor (the non-cuda-graph format) is rejected because the code needs to index positions 0/1/2.

Source

Thrown at python/sglang/srt/layers/attention/vision.py:486

        **kwargs,
    ) -> torch.Tensor:
        r"""
        Args:
            cu_seqlens: [b]
        Returns:
             [b * s, h, head_size]
        """
        if forward_metadata is not None:
            cu_seqlens_gpu = forward_metadata.cu_seqlens
            seq_lens = forward_metadata.seq_lens
            max_seqlen = forward_metadata.max_seqlen
            output = torch.empty_like(q)
        elif envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get():
            if "output_ws" not in kwargs:
                raise RuntimeError("output_ws should be prepared for cuda-graph mode")

            if not isinstance(cu_seqlens, list):
                raise RuntimeError("cuda-graph mode cu_seqlens should be a list")

            output = kwargs["output_ws"]
            cu_seqlens_gpu = cu_seqlens[0]
            seq_lens = cu_seqlens[1]
            max_seqlen = cu_seqlens[2]
        else:
            cu_seqlens_gpu = resolve_seqlens(cu_seqlens, bsz, seq_len, device=q.device)
            seq_lens = kwargs.get("sequence_lengths")
            if seq_lens is None:
                seq_lens = cu_seqlens_gpu[1:] - cu_seqlens_gpu[:-1]
            else:
                seq_lens = seq_lens.to(device=q.device, dtype=torch.int32)
            max_seqlen = resolve_precomputed_max_seqlen(
                cu_seqlens_gpu, kwargs.get("max_seqlen")
            )
            # [b * s, head, head_size]
            output = torch.empty_like(q)

View on GitHub (pinned to 0132848349)

Solutions

1. Pass a 3-element list: cu_seqlens=[cu_seqlens_gpu_tensor, seq_lens_tensor, max_seqlen_value] when cuda-graph mode is on.
2. Mirror how SGLang's own runner builds the argument during graph capture (bundle the three prepared tensors).
3. Disable SGLANG_VIT_ENABLE_CUDA_GRAPH if your caller cannot provide the bundled format.

Example fix

# before
attn(q, k, v, cu_seqlens=cu_seqlens_tensor)
# after
attn(q, k, v, cu_seqlens=[cu_seqlens_gpu, seq_lens, max_seqlen])

if envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get() and not isinstance(cu_seqlens, (list, tuple)) or len(cu_seqlens) != 3:
    cu_seqlens = [cu_seqlens_gpu, seq_lens, max_seqlen]  # bundle graph-stable tensors

def is_cg_cu_seqlens_flash3(x) -> bool:
    return isinstance(x, (list, tuple)) and len(x) == 3 and isinstance(x[0], torch.Tensor)

When it happens

Trigger: SGLANG_VIT_ENABLE_CUDA_GRAPH enabled and forward called with cu_seqlens as a torch.Tensor or other non-list (e.g. the plain cu_seqlens tensor used on the normal path) and forward_metadata None.

Common situations: Reusing a non-graph capture path's argument format when cuda-graph was enabled; custom vision-model integrations that pass only a cu_seqlens tensor; env var enabled globally while a model only supports the legacy call signature.

Understand the failure class

Background: "Wrong argument type", "must be a string", "expected Array or Prism::Scope": TypeError and ArgumentError when a library receives a value of the wrong type — this error's family across 28 libraries.

Related errors

• flashinfer_cudnn expects packed indptrs as a torch.Tensor
• output_ws should be prepared for cuda-graph mode
• actions must be a list[list[str]]
• Inkling reasoning_effort must not be a boolean
• SGLANG_DSA_TOPK_BROADCAST requires PyNCCL during CUDA graph

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0611056615dfd904. Report an issue: GitHub.