sgl-project/sglang · error · RuntimeError

Unsupported dtype {dtype}. Supported: float16, bfloat16, flo

Error message

Unsupported dtype {dtype}. Supported: float16, bfloat16, float32

What it means

The per-token FP8 quant kernel is JIT-compiled per input dtype via tvm_ffi; only float16, bfloat16 and float32 inputs have generated CUDA source. Any other dtype (fp8, int, fp64) hits this RuntimeError during module compilation lookup.

Source

Thrown at python/sglang/kernels/ops/quantization/per_token_quant_fp8.py:22

import torch

from sglang.kernels.jit.utils import (
    cache_once,
    get_jit_cuda_arch,
    load_jit,
    make_cpp_args,
)
from sglang.srt.utils.custom_op import register_custom_op

if TYPE_CHECKING:
    from tvm_ffi.module import Module


@cache_once
def _jit_per_token_quant_fp8_module(dtype: torch.dtype) -> Module:
    if dtype not in (torch.float16, torch.bfloat16, torch.float32):
        raise RuntimeError(
            f"Unsupported dtype {dtype}. Supported: float16, bfloat16, float32"
        )
    arch = get_jit_cuda_arch()
    use_fast_math = (arch.major, arch.minor) == (9, 0)
    math_mode = "fast_math" if use_fast_math else "precise_math"
    args = make_cpp_args(dtype)
    return load_jit(
        "per_token_quant_fp8",
        math_mode,
        *args,
        cuda_files=["gemm/per_token_quant_fp8.cuh"],
        cuda_wrappers=[("per_token_quant_fp8", f"per_token_quant_fp8<{args}>")],
        extra_cuda_cflags=["--use_fast_math"] if use_fast_math else [],
    )


@register_custom_op(
    op_name="per_token_quant_fp8",

View on GitHub (pinned to 0132848349)

Solutions

1. Cast input to float16/bfloat16/float32 before calling per_token_quant_fp8
2. Skip the call if the tensor is already fp8

Example fix

// before
q, s = per_token_quant_fp8(x_fp8)
// after
q, s = per_token_quant_fp8(x.to(torch.bfloat16))

if x.dtype not in (torch.float16, torch.bfloat16, torch.float32):
    x = x.to(torch.bfloat16)

def quantizable_dtype(t): return t.dtype in (torch.float16, torch.bfloat16, torch.float32)

When it happens

Trigger: Calling per_token_quant_fp8 with an input tensor whose dtype is not in {float16, bfloat16, float32}, e.g. already-quantized fp8 data or int32 activations.

Common situations: Double-quantizing (feeding an FP8 tensor back into the quant op), stray .double() casts in preprocessing, or test fixtures creating tensors with default dtypes.

Related errors

• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• int32-packed scale buffers require scale_ue8m0=True
• scale_ue8m0=True requires an int32-packed output_s
• Unsupported output_s dtype {output_s.dtype}
• Comfy full_precision_matrix_mult does not support fused line

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/51861bfcfdcfd86a. Report an issue: GitHub.