sgl-project/sglang · error · RuntimeError

output_ws should be prepared for cuda-graph mode

Error message

output_ws should be prepared for cuda-graph mode

What it means

When SGLANG_VIT_ENABLE_CUDA_GRAPH is enabled, the ViT forward path no longer allocates its own output tensor; it must write into a pre-allocated, graph-capture-stable output workspace passed via kwargs['output_ws']. The error is raised when forward() is called in cuda-graph mode without that key, because the output buffer address must be fixed across graph replays.

Source

Thrown at python/sglang/srt/layers/attention/vision.py:483

        seq_len: int,
        softmax_scale: Optional[float] = None,
        forward_metadata: Optional[VisionAttentionMetadata] = None,
        **kwargs,
    ) -> torch.Tensor:
        r"""
        Args:
            cu_seqlens: [b]
        Returns:
             [b * s, h, head_size]
        """
        if forward_metadata is not None:
            cu_seqlens_gpu = forward_metadata.cu_seqlens
            seq_lens = forward_metadata.seq_lens
            max_seqlen = forward_metadata.max_seqlen
            output = torch.empty_like(q)
        elif envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get():
            if "output_ws" not in kwargs:
                raise RuntimeError("output_ws should be prepared for cuda-graph mode")

            if not isinstance(cu_seqlens, list):
                raise RuntimeError("cuda-graph mode cu_seqlens should be a list")

            output = kwargs["output_ws"]
            cu_seqlens_gpu = cu_seqlens[0]
            seq_lens = cu_seqlens[1]
            max_seqlen = cu_seqlens[2]
        else:
            cu_seqlens_gpu = resolve_seqlens(cu_seqlens, bsz, seq_len, device=q.device)
            seq_lens = kwargs.get("sequence_lengths")
            if seq_lens is None:
                seq_lens = cu_seqlens_gpu[1:] - cu_seqlens_gpu[:-1]
            else:
                seq_lens = seq_lens.to(device=q.device, dtype=torch.int32)
            max_seqlen = resolve_precomputed_max_seqlen(
                cu_seqlens_gpu, kwargs.get("max_seqlen")
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Pass a pre-allocated output workspace: kwargs['output_ws'] = torch.empty_like(q) allocated once (not per step) so it stays stable across cuda-graph replays.
2. If you did not intend cuda-graph mode for the ViT, unset SGLANG_VIT_ENABLE_CUDA_GRAPH.
3. Ensure the caller uses the same prepared-metadata path (forward_metadata) the SGLang runner provides instead of calling forward raw.

Example fix

# before
out = vit_attn(q, k, v, cu_seqlens=cu_seqlens)
# after (cuda-graph mode)
out = vit_attn(q, k, v, cu_seqlens=cu_seqlens, output_ws=self._preallocated_output_ws)

use_cg = envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get()
if use_cg:
    if "output_ws" not in kwargs:
        kwargs = {**kwargs, "output_ws": self.vit_output_ws}  # pre-allocated torch.empty_like(q) buffer

def has_cuda_graph_ws(kwargs: dict, q: torch.Tensor) -> bool:
    ws = kwargs.get("output_ws")
    return isinstance(ws, torch.Tensor) and ws.shape == q.shape and ws.dtype == q.dtype

When it happens

Trigger: Calling VisionAttention.forward (flash3 path) with envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get() truthy, forward_metadata None, and no 'output_ws' entry in kwargs — typically a custom caller or a model integration that wasn't updated to pass the workspace.

Common situations: Enabling SGLANG_VIT_ENABLE_CUDA_GRAPH=1 on a new/patched vision model whose forward call site was not updated; upgrading SGLang where the ViT forward signature gained the output_ws requirement; running a multimodal model with an out-of-tree vision encoder wrapper.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• sequence_lengths should be prepared for vision flashinfer_cu
• cuda-graph mode cu_seqlens should be a list
• max_seqlen should be prepared for vision flashinfer_cudnn at
• Z-Image caption tensor must have rank 2 or 3
• mask_search_files_path_pos, mask_search_files_path_neg, and

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b95e7466d14fbbd1. Report an issue: GitHub.