{"record":{"id":"260af63e358a7d8e","repo":"sgl-project/sglang","slug":"visionflashinferattention-is-only-available-for-cu","errorCode":null,"errorMessage":"VisionFlashInferAttention is only available for cuda","messagePattern":"VisionFlashInferAttention is only available for cuda","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/vision.py","lineNumber":642,"sourceCode":"            v,\n            cu_seqlens_q=cu_seqlens_gpu,\n            cu_seqlens_k=cu_seqlens_gpu,\n            max_seqlen_q=max_seqlen,\n            max_seqlen_k=max_seqlen,\n            softmax_scale=softmax_scale,\n            ver=4,\n        )\n\n        return output\n\n\nclass VisionFlashInferAttention(nn.Module):\n    def __init__(\n        self,\n        **kwargs,\n    ):\n        if not _is_cuda:\n            raise Exception(\"VisionFlashInferAttention is only available for cuda\")\n        super().__init__()\n        self.workspace_buffer = (\n            kwargs[\"workspace_buffer\"] if \"workspace_buffer\" in kwargs else None\n        )\n\n    def forward(\n        self,\n        q: torch.Tensor,\n        k: torch.Tensor,\n        v: torch.Tensor,\n        cu_seqlens: torch.Tensor | SingletonCache | None,\n        bsz: int,\n        seq_len: int,\n        softmax_scale: Optional[float] = None,\n        forward_metadata: Optional[VisionAttentionMetadata] = None,\n        **kwargs,\n    ) -> torch.Tensor:\n        r\"\"\"","sourceCodeStart":624,"sourceCodeEnd":660,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/vision.py#L624-L660","documentation":"VisionFlashInferAttention wraps flashinfer kernels, which are compiled only for CUDA; __init__ raises immediately on non-CUDA builds (ROCm, MUSA, CPU) before setting up its workspace_buffer.","triggerScenarios":"Constructing VisionFlashInferAttention on a non-CUDA platform — typically because the vision encoder config selected the flashinfer backend.","commonSituations":"Serving multimodal models on ROCm with flashinfer selected as the vision attention backend; reusing a CUDA-oriented config/Docker image on other hardware; importing the class eagerly on CPU-only build machines.","solutions":["Choose a platform-supported vision attention implementation instead of flashinfer.","Confirm torch.version.cuda / CUDA availability before selecting the backend.","Only expose flashinfer as an option in CUDA build pipelines."],"exampleFix":"# before\nattn = VisionFlashInferAttention()\n# after\nimpl = \"flashinfer\" if _is_cuda else \"sdpa\"\nattn = VISION_ATTN_IMPLS[impl]()","handlingStrategy":"fallback","validationCode":"import torch\nfi_ok = torch.cuda.is_available() and getattr(torch.version, \"hip\", None) is None\nattn_impl = \"flashinfer\" if fi_ok else \"sdpa\"","typeGuard":"def supports_vision_flashinfer() -> bool:\n    import torch\n    return torch.cuda.is_available() and torch.version.hip is None","tryCatchPattern":"try:\n    attn = VisionFlashInferAttention(workspace_buffer=buf)\nexcept Exception:\n    attn = VisionSDPAAttention()","preventionTips":["Select vision attention backend per deployment target (CUDA-only for flashinfer).","Smoke-test backend construction in CI for each supported platform.","Keep config templates platform-parameterized rather than copy-pasted."],"tags":["sglang","vision-transformer","flashinfer","platform-support","hardware-compat"],"backgroundTag":"operation-not-supported-on-device","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}