{"record":{"id":"78896f9419c8daea","repo":"sgl-project/sglang","slug":"fp8-blockwise-scaled-mm-jit-kernel-requires-sm120","errorCode":null,"errorMessage":"fp8_blockwise_scaled_mm JIT kernel requires SM120 (Blackwell).","messagePattern":"fp8_blockwise_scaled_mm JIT kernel requires SM120 \\(Blackwell\\)\\.","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/gemm/fp8_blockwise_gemm.py","lineNumber":34,"sourceCode":"def _fp8_blockwise_cuda_flags() -> list[str]:\n    return [\n        \"-DNDEBUG\",\n        \"-DCUTE_USE_PACKED_TUPLE=1\",\n        \"-DCUTLASS_ENABLE_TENSOR_CORE_MMA=1\",\n        \"-DCUTLASS_VERSIONS_GENERATED\",\n        \"-DCUTLASS_TEST_LEVEL=0\",\n        \"-DCUTLASS_TEST_ENABLE_CACHED_RESULTS=1\",\n        \"-DCUTLASS_DEBUG_TRACE_LEVEL=0\",\n        \"--expt-relaxed-constexpr\",\n        \"--expt-extended-lambda\",\n    ]\n\n\n@cache_once\ndef _jit_fp8_blockwise_module() -> Module:\n    \"\"\"Compile and cache the SM120 fp8 blockwise GEMM module (handles fp16 + bf16).\"\"\"\n    if not is_sm120_supported():\n        raise RuntimeError(\n            \"fp8_blockwise_scaled_mm JIT kernel requires SM120 (Blackwell).\"\n        )\n    return load_jit(\n        \"fp8_blockwise_scaled_mm\",\n        cuda_files=[\"gemm/fp8_blockwise/fp8_blockwise_scaled_mm_entry.cuh\"],\n        cuda_wrappers=[\n            (\"fp8_blockwise_scaled_mm\", \"fp8_blockwise_scaled_mm\"),\n        ],\n        extra_dependencies=[\"cutlass\"],\n        extra_cuda_cflags=_fp8_blockwise_cuda_flags(),\n    )\n\n\n@register_custom_op(\n    op_name=\"fp8_blockwise_scaled_mm\",\n    mutates_args=[\"out\"],\n)\ndef _fp8_blockwise_scaled_mm_custom_op(","sourceCodeStart":16,"sourceCodeEnd":52,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/gemm/fp8_blockwise_gemm.py#L16-L52","documentation":"The fp8 blockwise-scaled GEMM JIT kernel is compiled specifically for SM120 (Blackwell consumer, e.g. RTX 5090) and refuses to build elsewhere, as tested by is_sm120_supported().","triggerScenarios":"Invoking _fp8_blockwise_scaled_mm_custom_op on SM90 (H100), SM100 (B200), or older GPUs — notably SM100 datacenter Blackwell is NOT SM120 and still fails.","commonSituations":"Assuming any Blackwell GPU works; deploying fp8 blockwise quantized models (DeepSeek-style) on Hopper or B200 where a different fp8 kernel must be used.","solutions":["Use the SM90/SM100-specific fp8 blockwise kernel variants for Hopper/B200.","Restrict this path to SM120 devices via is_sm120_supported().","Select the right kernel in the quantization backend based on get_device_sm()."],"exampleFix":"// before\nout = _fp8_blockwise_scaled_mm_custom_op(a, b, scales, out_dtype)\n// after\nif is_sm120_supported():\n    out = _fp8_blockwise_scaled_mm_custom_op(a, b, scales, out_dtype)\nelse:\n    out = fp8_blockwise_scaled_mm_dispatch(a, b, scales, out_dtype)  # backend per SM","handlingStrategy":"fallback","validationCode":"if not is_sm120_supported():\n    out = dispatch_fp8_blockwise_by_sm(a, b, scales, out_dtype)  # SM90/SM100 kernels","typeGuard":"def sm120() -> bool:\n    return torch.cuda.get_device_capability(0) == (12, 0)","tryCatchPattern":null,"preventionTips":["Remember SM100 (B200) is not SM120 — check both.","Select fp8 kernel per device architecture at init."],"tags":["gemm","fp8","blockwise","sm120","blackwell","gpu-architecture"],"backgroundTag":"gpu-architecture-not-supported","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}