sgl-project/sglang · error · ValueError

nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input

Error message

nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input and output only

What it means

The fused NVFP4 GEMM+SwiGLU+NVFP4-quant kernel quantizes its output back to FP4 for the down_proj GEMM, so both inputs and output must be float4_e2m1fn. Passing any other dtype string (fp8, bf16, fp16 output, etc.) selects a code path the kernel does not implement.

Source

Thrown at python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py:2861

) -> tuple[torch.Tensor, torch.Tensor]:
    """NVFP4 GEMM fused with SwiGLU and NVFP4 output quantization.

    Args:
        a: FP4-packed input activation, shape ``[M, K / 2]``.
        a_scale: Swizzled NVFP4 input scales,
            shape ``[round_up(M,128), round_up(K/16,4)]``.
        b: FP4-packed interleaved FC1 weight, shape ``[2 * I, K / 2]``.
        b_scale: Swizzled interleaved FC1 weight scales.
        alpha: GEMM global dequant scale, scalar or ``[1, 1]``.
        output_global_scale: Output quantization scale-up factor (= 1 /
            down_proj.input_scale_inv).
        enable_pdl: Enable Programmatic Dependent Launch for the fused kernel.

    Returns:
        ``(out_fp4, out_scale)`` directly consumable by the NVFP4 ``down_proj``.
    """
    if ab_dtype != "float4_e2m1fn" or c_dtype != "float4_e2m1fn":
        raise ValueError(
            "nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input "
            "and output only"
        )
    if a.device.type != "cuda" or b.device.type != "cuda":
        raise ValueError("nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors")

    major, minor = get_compute_capability(a.device)
    if major != 10:
        raise ValueError(
            f"nvfp4_gemm_swiglu_nvfp4_quant requires SM100, got SM{major}{minor}"
        )

    m = a.shape[0]
    k = a.shape[1] * 2
    n = b.shape[0]
    if b.shape[1] * 2 != k:
        raise ValueError(f"Shape mismatch: A K={k}, B K={b.shape[1] * 2}")
    if n % 2 != 0:

View on GitHub (pinned to 0132848349)

Solutions

1. Pass ab_dtype='float4_e2m1fn' and c_dtype='float4_e2m1fn'
2. If you need higher-precision output, use the non-fused gemm + swiglu + separate quant sequence instead

Example fix

// before
out, s = nvfp4_gemm_swiglu_nvfp4_quant(a, b, sf_a, sf_b, 'float4_e2m1fn', 'bfloat16')
// after
out, s = nvfp4_gemm_swiglu_nvfp4_quant(a, b, sf_a, sf_b, 'float4_e2m1fn', 'float4_e2m1fn')

assert ab_dtype == c_dtype == 'float4_e2m1fn'

def is_nvfp4_dtype_str(s): return s == 'float4_e2m1fn'

When it happens

Trigger: Calling nvfp4_gemm_swiglu_nvfp4_quant with ab_dtype or c_dtype not equal to 'float4_e2m1fn' — e.g. requesting a bf16 output for debugging or mixing with an FP8 config.

Common situations: Config typos in quant method names, using an fp8 block-quant config with this fused op, or older checkpoints saved before NVFP4 output support naming the dtype differently.

Related errors

• Type must match: {self.a_dtype} != {self.b_dtype}
• Shape mismatch: A K={k}, B K={b.shape[1] * 2}
• Output N={n_out} must be divisible by sf_vec_size={sf_vec_si
• Unsupported output_s dtype {output_s.dtype}
• Comfy full_precision_matrix_mult does not support fused line

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/74fe8969fc2e8ad1. Report an issue: GitHub.