{"record":{"id":"a4b9c57fc81f064f","repo":"sgl-project/sglang","slug":"nvfp4-global-scale-tensor-must-already-be-on-the-k","errorCode":null,"errorMessage":"NVFP4 global scale tensor must already be on the KV tensor device.","messagePattern":"NVFP4 global scale tensor must already be on the KV tensor device\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/kvfp4_tensor.py","lineNumber":201,"sourceCode":"\n        assert (\n            is_sm100_supported() or is_sm120_supported() or is_sm90_supported()\n        ), \"NVFP4 KV cache quantize requires SM100/SM120 or SM90 fallback GPU\"\n\n        b, m, n = tensor.shape\n        tensor_2d = tensor.reshape(b * m, n)\n\n        # The KV cache path passes preloaded per-layer scales already on device.\n        # Keep scalar/0-d support for tests and future fallback paths, but do not\n        # silently move tensor scales here.\n        if isinstance(global_scale, (int, float)):\n            global_scale = torch.tensor(\n                [global_scale], dtype=torch.float32, device=tensor.device\n            )\n        elif global_scale.dim() == 0:\n            global_scale = global_scale.unsqueeze(0)\n        elif global_scale.device != tensor.device:\n            raise ValueError(\n                \"NVFP4 global scale tensor must already be on the KV tensor device.\"\n            )\n\n        if is_sm100_supported() or is_sm120_supported():\n            from flashinfer import nvfp4_kv_quantize\n\n            # nvfp4_kv_quantize takes global_scale directly (not inverted)\n            fp4_2d, scales_2d = nvfp4_kv_quantize(tensor_2d, global_scale)\n        else:\n            # SM90: fp4_quantize takes inverted global_scale\n            from flashinfer import fp4_quantize\n\n            global_scale_inv = 1.0 / global_scale\n            fp4_2d, scales_2d = fp4_quantize(\n                tensor_2d,\n                global_scale_inv,\n                sf_vec_size=16,\n                sf_use_ue8m0=False,","sourceCodeStart":183,"sourceCodeEnd":219,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/kvfp4_tensor.py#L183-L219","documentation":"NVFP4 KV quantize accepts a global_scale that is a Python number, or a tensor already on the same device as the KV tensor. If a tensor is supplied and its device differs from tensor.device (e.g. scale still on CPU while KV states are on GPU), this ValueError is raised rather than silently triggering implicit cross-device copies in the quantization kernel path.","triggerScenarios":"Passing a torch tensor global_scale created on CPU (default device) while the hidden states / KV tensor lives on cuda:0; reusing a cached scale tensor after moving the model between devices.","commonSituations":"Integrating NVFP4 KV quantization in custom attention code; device migrations (model.to('cuda')) after scale tensors were captured; multi-GPU setups where scale was pinned to one rank's device.","solutions":["Pass a plain float/int for global_scale and let the function construct the tensor on tensor.device","Or explicitly move the scale: global_scale = global_scale.to(tensor.device) before calling quantize","Re-create scale tensors after any device move instead of caching stale ones"],"exampleFix":"# before\nscale = torch.tensor([2.0])  # CPU\nquant_tensor = quantize(kv_tensor, scale)\n# after\nquant_tensor = quantize(kv_tensor, 2.0)  # or scale.to(kv_tensor.device)","handlingStrategy":"validation","validationCode":"if isinstance(global_scale, torch.Tensor):\n    assert global_scale.device == tensor.device, \"scale device mismatch\"\n    if global_scale.dim() == 0:\n        global_scale = global_scale.unsqueeze(0)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pass floats for global_scale when possible","After any model.to(device), refresh cached scale tensors"],"tags":["nvfp4","kv-cache","device-mismatch","quantization"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}