{"record":{"id":"74fe8969fc2e8ad1","repo":"sgl-project/sglang","slug":"nvfp4-gemm-swiglu-nvfp4-quant-currently-supports-n","errorCode":null,"errorMessage":"nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input and output only","messagePattern":"nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input and output only","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py","lineNumber":2861,"sourceCode":") -> tuple[torch.Tensor, torch.Tensor]:\n    \"\"\"NVFP4 GEMM fused with SwiGLU and NVFP4 output quantization.\n\n    Args:\n        a: FP4-packed input activation, shape ``[M, K / 2]``.\n        a_scale: Swizzled NVFP4 input scales,\n            shape ``[round_up(M,128), round_up(K/16,4)]``.\n        b: FP4-packed interleaved FC1 weight, shape ``[2 * I, K / 2]``.\n        b_scale: Swizzled interleaved FC1 weight scales.\n        alpha: GEMM global dequant scale, scalar or ``[1, 1]``.\n        output_global_scale: Output quantization scale-up factor (= 1 /\n            down_proj.input_scale_inv).\n        enable_pdl: Enable Programmatic Dependent Launch for the fused kernel.\n\n    Returns:\n        ``(out_fp4, out_scale)`` directly consumable by the NVFP4 ``down_proj``.\n    \"\"\"\n    if ab_dtype != \"float4_e2m1fn\" or c_dtype != \"float4_e2m1fn\":\n        raise ValueError(\n            \"nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input \"\n            \"and output only\"\n        )\n    if a.device.type != \"cuda\" or b.device.type != \"cuda\":\n        raise ValueError(\"nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors\")\n\n    major, minor = get_compute_capability(a.device)\n    if major != 10:\n        raise ValueError(\n            f\"nvfp4_gemm_swiglu_nvfp4_quant requires SM100, got SM{major}{minor}\"\n        )\n\n    m = a.shape[0]\n    k = a.shape[1] * 2\n    n = b.shape[0]\n    if b.shape[1] * 2 != k:\n        raise ValueError(f\"Shape mismatch: A K={k}, B K={b.shape[1] * 2}\")\n    if n % 2 != 0:","sourceCodeStart":2843,"sourceCodeEnd":2879,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py#L2843-L2879","documentation":"The fused NVFP4 GEMM+SwiGLU+NVFP4-quant kernel quantizes its output back to FP4 for the down_proj GEMM, so both inputs and output must be float4_e2m1fn. Passing any other dtype string (fp8, bf16, fp16 output, etc.) selects a code path the kernel does not implement.","triggerScenarios":"Calling nvfp4_gemm_swiglu_nvfp4_quant with ab_dtype or c_dtype not equal to 'float4_e2m1fn' — e.g. requesting a bf16 output for debugging or mixing with an FP8 config.","commonSituations":"Config typos in quant method names, using an fp8 block-quant config with this fused op, or older checkpoints saved before NVFP4 output support naming the dtype differently.","solutions":["Pass ab_dtype='float4_e2m1fn' and c_dtype='float4_e2m1fn'","If you need higher-precision output, use the non-fused gemm + swiglu + separate quant sequence instead"],"exampleFix":"// before\nout, s = nvfp4_gemm_swiglu_nvfp4_quant(a, b, sf_a, sf_b, 'float4_e2m1fn', 'bfloat16')\n// after\nout, s = nvfp4_gemm_swiglu_nvfp4_quant(a, b, sf_a, sf_b, 'float4_e2m1fn', 'float4_e2m1fn')","handlingStrategy":"validation","validationCode":"assert ab_dtype == c_dtype == 'float4_e2m1fn'","typeGuard":"def is_nvfp4_dtype_str(s): return s == 'float4_e2m1fn'","tryCatchPattern":null,"preventionTips":["Centralize dtype strings as constants instead of literals"],"tags":["nvfp4","dtype-validation","quantization"],"backgroundTag":"dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}