{"record":{"id":"e4f62ed275011811","repo":"sgl-project/sglang","slug":"nvfp4-gemm-swiglu-nvfp4-quant-requires-cuda-tensor","errorCode":null,"errorMessage":"nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors","messagePattern":"nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py","lineNumber":2866,"sourceCode":"        a_scale: Swizzled NVFP4 input scales,\n            shape ``[round_up(M,128), round_up(K/16,4)]``.\n        b: FP4-packed interleaved FC1 weight, shape ``[2 * I, K / 2]``.\n        b_scale: Swizzled interleaved FC1 weight scales.\n        alpha: GEMM global dequant scale, scalar or ``[1, 1]``.\n        output_global_scale: Output quantization scale-up factor (= 1 /\n            down_proj.input_scale_inv).\n        enable_pdl: Enable Programmatic Dependent Launch for the fused kernel.\n\n    Returns:\n        ``(out_fp4, out_scale)`` directly consumable by the NVFP4 ``down_proj``.\n    \"\"\"\n    if ab_dtype != \"float4_e2m1fn\" or c_dtype != \"float4_e2m1fn\":\n        raise ValueError(\n            \"nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input \"\n            \"and output only\"\n        )\n    if a.device.type != \"cuda\" or b.device.type != \"cuda\":\n        raise ValueError(\"nvfp4_gemm_swiglu_nvfp4_quant requires CUDA tensors\")\n\n    major, minor = get_compute_capability(a.device)\n    if major != 10:\n        raise ValueError(\n            f\"nvfp4_gemm_swiglu_nvfp4_quant requires SM100, got SM{major}{minor}\"\n        )\n\n    m = a.shape[0]\n    k = a.shape[1] * 2\n    n = b.shape[0]\n    if b.shape[1] * 2 != k:\n        raise ValueError(f\"Shape mismatch: A K={k}, B K={b.shape[1] * 2}\")\n    if n % 2 != 0:\n        raise ValueError(f\"Interleaved FC1 N must be even, got {n}\")\n\n    l = 1\n    n_out = n // 2\n    if n_out % sf_vec_size != 0:","sourceCodeStart":2848,"sourceCodeEnd":2884,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py#L2848-L2884","documentation":"The fused kernel is implemented only as CUDA (SM100) CUTLASS/TVM-FFI code and does not compile or dispatch for CPU or other backends. Both the activation a and weight b must already live on a CUDA device when the op is called.","triggerScenarios":"Calling nvfp4_gemm_swiglu_nvfp4_quant with tensors on CPU (e.g. before .cuda(), or in a CPU-only test environment), or with a on GPU and b on CPU.","commonSituations":"Unit tests without GPU, weight tensors not moved to device in process_weights_after_loading, or CUDA_VISIBLE_DEVICES misconfiguration leaving tensors on CPU.","solutions":["Move both a and b (and scale tensors) to the CUDA device with .to('cuda') before the call","Add a guard in test code to skip when not torch.cuda.is_available()"],"exampleFix":"// before\nout = nvfp4_gemm_swiglu_nvfp4_quant(a, b, ...)\n// after\nout = nvfp4_gemm_swiglu_nvfp4_quant(a.cuda(), b.cuda(), ...)","handlingStrategy":"validation","validationCode":"assert a.is_cuda and b.is_cuda","typeGuard":"def both_cuda(*ts): return all(t.is_cuda for t in ts)","tryCatchPattern":null,"preventionTips":["Move model shards to device in process_weights_after_loading","Skip fused-kernel tests without CUDA"],"tags":["nvfp4","cuda","device-placement","gpu-kernel"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}