sgl-project/sglang · error · RuntimeError

fp8_blockwise_scaled_mm JIT kernel requires SM120 (Blackwell

Error message

fp8_blockwise_scaled_mm JIT kernel requires SM120 (Blackwell).

What it means

The fp8 blockwise-scaled GEMM JIT kernel is compiled specifically for SM120 (Blackwell consumer, e.g. RTX 5090) and refuses to build elsewhere, as tested by is_sm120_supported().

Source

Thrown at python/sglang/kernels/ops/gemm/fp8_blockwise_gemm.py:34

def _fp8_blockwise_cuda_flags() -> list[str]:
    return [
        "-DNDEBUG",
        "-DCUTE_USE_PACKED_TUPLE=1",
        "-DCUTLASS_ENABLE_TENSOR_CORE_MMA=1",
        "-DCUTLASS_VERSIONS_GENERATED",
        "-DCUTLASS_TEST_LEVEL=0",
        "-DCUTLASS_TEST_ENABLE_CACHED_RESULTS=1",
        "-DCUTLASS_DEBUG_TRACE_LEVEL=0",
        "--expt-relaxed-constexpr",
        "--expt-extended-lambda",
    ]


@cache_once
def _jit_fp8_blockwise_module() -> Module:
    """Compile and cache the SM120 fp8 blockwise GEMM module (handles fp16 + bf16)."""
    if not is_sm120_supported():
        raise RuntimeError(
            "fp8_blockwise_scaled_mm JIT kernel requires SM120 (Blackwell)."
        )
    return load_jit(
        "fp8_blockwise_scaled_mm",
        cuda_files=["gemm/fp8_blockwise/fp8_blockwise_scaled_mm_entry.cuh"],
        cuda_wrappers=[
            ("fp8_blockwise_scaled_mm", "fp8_blockwise_scaled_mm"),
        ],
        extra_dependencies=["cutlass"],
        extra_cuda_cflags=_fp8_blockwise_cuda_flags(),
    )


@register_custom_op(
    op_name="fp8_blockwise_scaled_mm",
    mutates_args=["out"],
)
def _fp8_blockwise_scaled_mm_custom_op(

View on GitHub (pinned to 0132848349)

Solutions

1. Use the SM90/SM100-specific fp8 blockwise kernel variants for Hopper/B200.
2. Restrict this path to SM120 devices via is_sm120_supported().
3. Select the right kernel in the quantization backend based on get_device_sm().

Example fix

// before
out = _fp8_blockwise_scaled_mm_custom_op(a, b, scales, out_dtype)
// after
if is_sm120_supported():
    out = _fp8_blockwise_scaled_mm_custom_op(a, b, scales, out_dtype)
else:
    out = fp8_blockwise_scaled_mm_dispatch(a, b, scales, out_dtype)  # backend per SM

if not is_sm120_supported():
    out = dispatch_fp8_blockwise_by_sm(a, b, scales, out_dtype)  # SM90/SM100 kernels

def sm120() -> bool:
    return torch.cuda.get_device_capability(0) == (12, 0)

When it happens

Trigger: Invoking _fp8_blockwise_scaled_mm_custom_op on SM90 (H100), SM100 (B200), or older GPUs — notably SM100 datacenter Blackwell is NOT SM120 and still fails.

Common situations: Assuming any Blackwell GPU works; deploying fp8 blockwise quantized models (DeepSeek-style) on Hopper or B200 where a different fp8 kernel must be used.

Related errors

• cutedsl_bf16_gemm requires an SM10x GPU
• (head_dim, head_dim_v)=({head_dim}, {head_dim_v}) exceeds SM
• flashinfer_sparse_mla supports only GLM DSA with FP8 KV cach
• GLM DSA with FP8 KV cache on NVIDIA SM120/SM121 supports onl
• cta_n={cta_n} invalid for use_2cta={use_2cta}: bf16 K-major

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/78896f9419c8daea. Report an issue: GitHub.