sgl-project/sglang · error · Exception

VisionFlashInferAttention is only available for cuda

Error message

VisionFlashInferAttention is only available for cuda

What it means

VisionFlashInferAttention wraps flashinfer kernels, which are compiled only for CUDA; __init__ raises immediately on non-CUDA builds (ROCm, MUSA, CPU) before setting up its workspace_buffer.

Source

Thrown at python/sglang/srt/layers/attention/vision.py:642

            v,
            cu_seqlens_q=cu_seqlens_gpu,
            cu_seqlens_k=cu_seqlens_gpu,
            max_seqlen_q=max_seqlen,
            max_seqlen_k=max_seqlen,
            softmax_scale=softmax_scale,
            ver=4,
        )

        return output


class VisionFlashInferAttention(nn.Module):
    def __init__(
        self,
        **kwargs,
    ):
        if not _is_cuda:
            raise Exception("VisionFlashInferAttention is only available for cuda")
        super().__init__()
        self.workspace_buffer = (
            kwargs["workspace_buffer"] if "workspace_buffer" in kwargs else None
        )

    def forward(
        self,
        q: torch.Tensor,
        k: torch.Tensor,
        v: torch.Tensor,
        cu_seqlens: torch.Tensor | SingletonCache | None,
        bsz: int,
        seq_len: int,
        softmax_scale: Optional[float] = None,
        forward_metadata: Optional[VisionAttentionMetadata] = None,
        **kwargs,
    ) -> torch.Tensor:
        r"""

View on GitHub (pinned to 0132848349)

Solutions

Choose a platform-supported vision attention implementation instead of flashinfer.
Confirm torch.version.cuda / CUDA availability before selecting the backend.
Only expose flashinfer as an option in CUDA build pipelines.

Example fix

# before
attn = VisionFlashInferAttention()
# after
impl = "flashinfer" if _is_cuda else "sdpa"
attn = VISION_ATTN_IMPLS[impl]()

Defensive patterns

Strategy: fallback

Validate before calling

import torch
fi_ok = torch.cuda.is_available() and getattr(torch.version, "hip", None) is None
attn_impl = "flashinfer" if fi_ok else "sdpa"

Type guard

def supports_vision_flashinfer() -> bool:
    import torch
    return torch.cuda.is_available() and torch.version.hip is None

Try / catch

try:
    attn = VisionFlashInferAttention(workspace_buffer=buf)
except Exception:
    attn = VisionSDPAAttention()

Prevention

Select vision attention backend per deployment target (CUDA-only for flashinfer).
Smoke-test backend construction in CI for each supported platform.
Keep config templates platform-parameterized rather than copy-pasted.

When it happens

Trigger: Constructing VisionFlashInferAttention on a non-CUDA platform — typically because the vision encoder config selected the flashinfer backend.

Common situations: Serving multimodal models on ROCm with flashinfer selected as the vision attention backend; reusing a CUDA-oriented config/Docker image on other hardware; importing the class eagerly on CPU-only build machines.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/260af63e358a7d8e. Report an issue: GitHub.