sgl-project/sglang · error · ValueError

nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors

Error message

nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors

What it means

The fused kernel is implemented only as CUDA (SM100) CUTLASS/TVM-FFI code and does not compile or dispatch for CPU or other backends. Both the activation a and weight b must already live on a CUDA device when the op is called.

Source

Thrown at python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py:2866

        a_scale: Swizzled NVFP4 input scales,
            shape ``[round_up(M,128), round_up(K/16,4)]``.
        b: FP4-packed interleaved FC1 weight, shape ``[2 * I, K / 2]``.
        b_scale: Swizzled interleaved FC1 weight scales.
        alpha: GEMM global dequant scale, scalar or ``[1, 1]``.
        output_global_scale: Output quantization scale-up factor (= 1 /
            down_proj.input_scale_inv).
        enable_pdl: Enable Programmatic Dependent Launch for the fused kernel.

    Returns:
        ``(out_fp4, out_scale)`` directly consumable by the NVFP4 ``down_proj``.
    """
    if ab_dtype != "float4_e2m1fn" or c_dtype != "float4_e2m1fn":
        raise ValueError(
            "nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input "
            "and output only"
        )
    if a.device.type != "cuda" or b.device.type != "cuda":
        raise ValueError("nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors")

    major, minor = get_compute_capability(a.device)
    if major != 10:
        raise ValueError(
            f"nvfp4_gemm_swiglu_nvfp4_quant requires SM100, got SM{major}{minor}"
        )

    m = a.shape[0]
    k = a.shape[1] * 2
    n = b.shape[0]
    if b.shape[1] * 2 != k:
        raise ValueError(f"Shape mismatch: A K={k}, B K={b.shape[1] * 2}")
    if n % 2 != 0:
        raise ValueError(f"Interleaved FC1 N must be even, got {n}")

    l = 1
    n_out = n // 2
    if n_out % sf_vec_size != 0:

View on GitHub (pinned to 0132848349)

Solutions

1. Move both a and b (and scale tensors) to the CUDA device with .to('cuda') before the call
2. Add a guard in test code to skip when not torch.cuda.is_available()

Example fix

// before
out = nvfp4_gemm_swiglu_nvfp4_quant(a, b, ...)
// after
out = nvfp4_gemm_swiglu_nvfp4_quant(a.cuda(), b.cuda(), ...)

assert a.is_cuda and b.is_cuda

def both_cuda(*ts): return all(t.is_cuda for t in ts)

When it happens

Trigger: Calling nvfp4_gemm_swiglu_nvfp4_quant with tensors on CPU (e.g. before .cuda(), or in a CPU-only test environment), or with a on GPU and b on CPU.

Common situations: Unit tests without GPU, weight tensors not moved to device in process_weights_after_loading, or CUDA_VISIBLE_DEVICES misconfiguration leaving tensors on CPU.

Related errors

• cos and sin must be CUDA and last-dim contiguous
• Type must match: {self.a_dtype} != {self.b_dtype}
• nvfp4_gemm_swiglu_nvfp4_quant requires SM100, got SM{major}{
• renorm kernels require a CUDA/HIP tensor
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e4f62ed275011811. Report an issue: GitHub.