sgl-project/sglang · error · RuntimeError

qprep_bf16_fp8_sm90 requires an SM90 (Hopper) GPU

Error message

qprep_bf16_fp8_sm90 requires an SM90 (Hopper) GPU

What it means

qprep_bf16_fp8_sm90's JIT module is gated on the GPU architecture: its CUDA source uses SM90 (Hopper) features, so _jit_qprep_bf16_fp8_module raises RuntimeError when torch.cuda.get_device_capability()[0] != 9. This triggers lazily on the first call via q8kv8_qprep_fwd.

Source

Thrown at python/sglang/kernels/ops/attention/qprep_bf16_fp8_sm90.py:31

from typing import TYPE_CHECKING

import torch

from sglang.kernels.jit.utils import cache_once, load_jit
from sglang.kernels.kernel_api_logging import debug_kernel_api

if TYPE_CHECKING:
    from tvm_ffi.module import Module


N_LORA = 512  # kv_lora_rank (nope output dim)
ROPE_DIM = 64  # qk_rope_head_dim


@cache_once
def _jit_qprep_bf16_fp8_module() -> Module:
    if torch.cuda.get_device_capability()[0] != 9:
        raise RuntimeError("qprep_bf16_fp8_sm90 requires an SM90 (Hopper) GPU")
    return load_jit(
        "qprep_bf16_fp8_sm90",
        cuda_files=["qprep_bf16_fp8_sm90/entry.cuh"],
        cuda_wrappers=[("dispatch", "qprep_bf16_fp8_dispatch")],
        # Same minimal flag set as the sparse_mla_q8kv8_prefill_sm90 JIT
        # build (per-flag ablation there showed the rest are no-ops).
        extra_cuda_cflags=[
            "-O3",
            "-DNDEBUG",
            "-DCUTE_USE_PACKED_TUPLE=1",
            "-DCUTLASS_ENABLE_TENSOR_CORE_MMA=1",
            "--use_fast_math",
        ],
        extra_dependencies=["cutlass"],
    )


# torch._C._cuda_getCurrentRawStream returns the cudaStream_t pointer expected

View on GitHub (pinned to 0132848349)

Solutions

1. Run on an H100/H200 (SM90) GPU
2. Disable the bf16->fp8 qprep path / select a different quantized prefill kernel on non-SM90 hardware
3. Gate backend selection on the device capability before the first forward

Example fix

# before
out = q8kv8_qprep_fwd(q, ...)  # on A100 -> RuntimeError
# after
if torch.cuda.get_device_capability()[0] == 9:
    out = q8kv8_qprep_fwd(q, ...)
else:
    out = fallback_qprep(q, ...)

if torch.cuda.get_device_capability()[0] != 9:
    qprep = fallback_qprep  # non-SM90 path
else:
    from ... import q8kv8_qprep_fwd as qprep

except RuntimeError as e: if 'SM90' in str(e): switch to non-quantized/fallback prefill

When it happens

Trigger: Calling q8kv8_qprep_fwd on any non-Hopper GPU (A100 SM80, Blackwell SM100, or older) which invokes the @cache_once JIT loader.

Common situations: Running the q8kv8 sparse prefill pipeline on A100/Ada/Blackwell machines; CI runners without H100; version upgrades enabling this qprep path by default on heterogeneous clusters.

Related errors

• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `
• flash_attn at sgl-kernel is only supported on sm90 and above
• Flash attention currently only supported for compute capabil
• dsv3_fused_a_gemm requires SM90 (Hopper) or later
• Cannot find CUTLASS headers required for JIT compilation. Pl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b89ec25e68450cca. Report an issue: GitHub.