sgl-project/sglang · critical · ImportError

Can not import FA3 in sgl_kernel. Please check your installa

Error message

Can not import FA3 in sgl_kernel. Please check your installation.

What it means

Module-level import guard in sgl_kernel/flash_attn.py: if `from sgl_kernel import flash_ops` fails for any reason, the module raises ImportError claiming FA3 cannot be imported. The bare except means the real cause (missing CUDA extension, libtorch ABI mismatch, missing libflashinfer/sgl shared libs) is hidden inside the chained exception.

Source

Thrown at python/sglang/kernels/aot/python/sgl_kernel/flash_attn.py:10

from functools import lru_cache
from typing import Optional, Union

import torch
from sgl_kernel.debug_utils import maybe_wrap_debug_kernel

try:
    from sgl_kernel import flash_ops
except:
    raise ImportError(
        "Can not import FA3 in sgl_kernel. Please check your installation."
    )


@lru_cache(maxsize=1)
def is_fa3_supported(device=None) -> bool:
    #  There some fa3 FYI
    #  FA3 can fail without a enough shared memory for a some shapes, such as higher
    #  hidden_dim or some special cases.
    #  Right now, fa3 is supported for sm80/sm87 and sm86/sm89. The main different
    #  Between sm80/sm87 and sm86/sm89 is the shared memory size. you can follow the link below for more information
    #  https://docs.nvidia.com/cuda/cuda-c-programming-guide/#shared-memory-8-x
    #  And for sgl-kernel right now, we can build fa3 on sm80/sm86/sm89/sm90a.
    #  That means if you use A100/A*0/L20/L40/L40s/4090 you can use fa3.
    return (torch.version.cuda >= "12.3") and (
        torch.cuda.get_device_capability(device)[0] == 9
        or torch.cuda.get_device_capability(device)[0] == 8
    )

View on GitHub (pinned to 0132848349)

Solutions

1. Reinstall sgl-kernel matching your torch/CUDA: `pip install -U sgl-kernel --force-reinstall` (pick the CUDA-specific wheel index if needed).
2. Check `python -c "import sgl_kernel"` to surface the real underlying error (the bare except hides it).
3. Verify torch version and CUDA runtime match the wheel's build (see sgl-kernel release notes for the torch/CUDA matrix).
4. Ensure LD_LIBRARY_PATH includes the sgl_kernel package dir if libs fail to resolve.

Example fix

# before
pip install sgl-kernel  # may pull mismatched torch build
# after
pip install sgl-kernel --force-reinstall --no-deps \
  --index-url https://flashinfer.ai/whl/cu124/torch2.5  # match your env

try:
    import sgl_kernel.flash_ops  # noqa: F401
    fa3_available = True
except Exception:
    fa3_available = False

def fa3_available() -> bool:
    try:
        import sgl_kernel.flash_ops  # noqa: F401
        return True
    except Exception:
        return False

try:
    from sgl_kernel.flash_attn import flash_attn_with_kvcache
except ImportError as e:
    if "FA3" in str(e):
        flash_attn_with_kvcache = None  # select triton/flashinfer backend instead

When it happens

Trigger: importing sgl_kernel.flash_attn when flash_ops (the compiled FA3 ops) failed to load — broken sgl-kernel wheel, CUDA extension not built for your torch/CUDA version, missing LD_LIBRARY_PATH for bundled .so files; any code path importing flash_attn (FA3 backend init, is_fa3_supported()).

Common situations: sgl-kernel version mismatched with installed torch; pip installed a wheel built for a different CUDA version; container missing GPU libs; upgrading torch without upgrading sgl-kernel.

Related errors

• Failed to load serve backend {name!r} from {self._entry_poin
• v_cache must be provided
• k_cache can only be None when only_qv=True
• q can only be None when only_qv=True
• q must be provided unless qv is provided with only_qv=True

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8930f3f85994845f. Report an issue: GitHub.