{"record":{"id":"b95e7466d14fbbd1","repo":"sgl-project/sglang","slug":"output-ws-should-be-prepared-for-cuda-graph-mode","errorCode":null,"errorMessage":"output_ws should be prepared for cuda-graph mode","messagePattern":"output_ws should be prepared for cuda-graph mode","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/vision.py","lineNumber":483,"sourceCode":"        seq_len: int,\n        softmax_scale: Optional[float] = None,\n        forward_metadata: Optional[VisionAttentionMetadata] = None,\n        **kwargs,\n    ) -> torch.Tensor:\n        r\"\"\"\n        Args:\n            cu_seqlens: [b]\n        Returns:\n             [b * s, h, head_size]\n        \"\"\"\n        if forward_metadata is not None:\n            cu_seqlens_gpu = forward_metadata.cu_seqlens\n            seq_lens = forward_metadata.seq_lens\n            max_seqlen = forward_metadata.max_seqlen\n            output = torch.empty_like(q)\n        elif envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get():\n            if \"output_ws\" not in kwargs:\n                raise RuntimeError(\"output_ws should be prepared for cuda-graph mode\")\n\n            if not isinstance(cu_seqlens, list):\n                raise RuntimeError(\"cuda-graph mode cu_seqlens should be a list\")\n\n            output = kwargs[\"output_ws\"]\n            cu_seqlens_gpu = cu_seqlens[0]\n            seq_lens = cu_seqlens[1]\n            max_seqlen = cu_seqlens[2]\n        else:\n            cu_seqlens_gpu = resolve_seqlens(cu_seqlens, bsz, seq_len, device=q.device)\n            seq_lens = kwargs.get(\"sequence_lengths\")\n            if seq_lens is None:\n                seq_lens = cu_seqlens_gpu[1:] - cu_seqlens_gpu[:-1]\n            else:\n                seq_lens = seq_lens.to(device=q.device, dtype=torch.int32)\n            max_seqlen = resolve_precomputed_max_seqlen(\n                cu_seqlens_gpu, kwargs.get(\"max_seqlen\")\n            )","sourceCodeStart":465,"sourceCodeEnd":501,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/vision.py#L465-L501","documentation":"When SGLANG_VIT_ENABLE_CUDA_GRAPH is enabled, the ViT forward path no longer allocates its own output tensor; it must write into a pre-allocated, graph-capture-stable output workspace passed via kwargs['output_ws']. The error is raised when forward() is called in cuda-graph mode without that key, because the output buffer address must be fixed across graph replays.","triggerScenarios":"Calling VisionAttention.forward (flash3 path) with envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get() truthy, forward_metadata None, and no 'output_ws' entry in kwargs — typically a custom caller or a model integration that wasn't updated to pass the workspace.","commonSituations":"Enabling SGLANG_VIT_ENABLE_CUDA_GRAPH=1 on a new/patched vision model whose forward call site was not updated; upgrading SGLang where the ViT forward signature gained the output_ws requirement; running a multimodal model with an out-of-tree vision encoder wrapper.","solutions":["Pass a pre-allocated output workspace: kwargs['output_ws'] = torch.empty_like(q) allocated once (not per step) so it stays stable across cuda-graph replays.","If you did not intend cuda-graph mode for the ViT, unset SGLANG_VIT_ENABLE_CUDA_GRAPH.","Ensure the caller uses the same prepared-metadata path (forward_metadata) the SGLang runner provides instead of calling forward raw."],"exampleFix":"# before\nout = vit_attn(q, k, v, cu_seqlens=cu_seqlens)\n# after (cuda-graph mode)\nout = vit_attn(q, k, v, cu_seqlens=cu_seqlens, output_ws=self._preallocated_output_ws)","handlingStrategy":"validation","validationCode":"use_cg = envs.SGLANG_VIT_ENABLE_CUDA_GRAPH.get()\nif use_cg:\n    if \"output_ws\" not in kwargs:\n        kwargs = {**kwargs, \"output_ws\": self.vit_output_ws}  # pre-allocated torch.empty_like(q) buffer","typeGuard":"def has_cuda_graph_ws(kwargs: dict, q: torch.Tensor) -> bool:\n    ws = kwargs.get(\"output_ws\")\n    return isinstance(ws, torch.Tensor) and ws.shape == q.shape and ws.dtype == q.dtype","tryCatchPattern":null,"preventionTips":["Allocate output workspaces once per shape at model init, not per forward, so cuda-graph replays see stable addresses.","Gate on the env var in your wrapper and translate formats centrally.","Add unit tests that run the ViT forward with SGLANG_VIT_ENABLE_CUDA_GRAPH=1."],"tags":["sglang","vision-transformer","cuda-graph","kwargs-validation","multimodal"],"backgroundTag":"missing-required-argument","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}