sgl-project/sglang · error · ValueError
NVFP4 global scale tensor must already be on the KV tensor…
Error message
NVFP4 global scale tensor must already be on the KV tensor device.
What it means
NVFP4 KV quantize accepts a global_scale that is a Python number, or a tensor already on the same device as the KV tensor. If a tensor is supplied and its device differs from tensor.device (e.g. scale still on CPU while KV states are on GPU), this ValueError is raised rather than silently triggering implicit cross-device copies in the quantization kernel path.
Solutions
- Pass a plain float/int for global_scale and let the function construct the tensor on tensor.device
- Or explicitly move the scale: global_scale = global_scale.to(tensor.device) before calling quantize
- Re-create scale tensors after any device move instead of caching stale ones
Example fix
# before scale = torch.tensor([2.0]) # CPU quant_tensor = quantize(kv_tensor, scale) # after quant_tensor = quantize(kv_tensor, 2.0) # or scale.to(kv_tensor.device)
Defensive patterns
Strategy: validation
Validate before calling
if isinstance(global_scale, torch.Tensor):
assert global_scale.device == tensor.device, "scale device mismatch"
if global_scale.dim() == 0:
global_scale = global_scale.unsqueeze(0) Prevention
- Pass floats for global_scale when possible
- After any model.to(device), refresh cached scale tensors
When it happens
Trigger: Passing a torch tensor global_scale created on CPU (default device) while the hidden states / KV tensor lives on cuda:0; reusing a cached scale tensor after moving the model between devices.
Common situations: Integrating NVFP4 KV quantization in custom attention code; device migrations (model.to('cuda')) after scale tensors were captured; multi-GPU setups where scale was pinned to one rank's device.
Related errors
- Ascend A2/A3 NPU does not support nvfp4…
- Comfy full_precision_matrix_mult does not support fused…
- comfy_nvfp4 is inferred from per-layer checkpoint metadata…
- Comfy NVFP4 layer has an incompatible pre_quant_scale
- Comfy NVFP4 layer has incompatible weight/scale shapes: and
AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28).
Data as JSON: /api/errors/a4b9c57fc81f064f.
Report an issue: GitHub.
Appendix: source
Thrown at python/sglang/srt/layers/quantization/kvfp4_tensor.py:201
assert (
is_sm100_supported() or is_sm120_supported() or is_sm90_supported()
), "NVFP4 KV cache quantize requires SM100/SM120 or SM90 fallback GPU"
b, m, n = tensor.shape
tensor_2d = tensor.reshape(b * m, n)
# The KV cache path passes preloaded per-layer scales already on device.
# Keep scalar/0-d support for tests and future fallback paths, but do not
# silently move tensor scales here.
if isinstance(global_scale, (int, float)):
global_scale = torch.tensor(
[global_scale], dtype=torch.float32, device=tensor.device
)
elif global_scale.dim() == 0:
global_scale = global_scale.unsqueeze(0)
elif global_scale.device != tensor.device:
raise ValueError(
"NVFP4 global scale tensor must already be on the KV tensor device."
)
if is_sm100_supported() or is_sm120_supported():
from flashinfer import nvfp4_kv_quantize
# nvfp4_kv_quantize takes global_scale directly (not inverted)
fp4_2d, scales_2d = nvfp4_kv_quantize(tensor_2d, global_scale)
else:
# SM90: fp4_quantize takes inverted global_scale
from flashinfer import fp4_quantize
global_scale_inv = 1.0 / global_scale
fp4_2d, scales_2d = fp4_quantize(
tensor_2d,
global_scale_inv,
sf_vec_size=16,
sf_use_ue8m0=False,View on GitHub (pinned to 0132848349)