sgl-project/sglang · error · ValueError

NVFP4 global scale tensor must already be on the KV tensor d

Error message

NVFP4 global scale tensor must already be on the KV tensor device.

What it means

NVFP4 KV quantize accepts a global_scale that is a Python number, or a tensor already on the same device as the KV tensor. If a tensor is supplied and its device differs from tensor.device (e.g. scale still on CPU while KV states are on GPU), this ValueError is raised rather than silently triggering implicit cross-device copies in the quantization kernel path.

Source

Thrown at python/sglang/srt/layers/quantization/kvfp4_tensor.py:201

        assert (
            is_sm100_supported() or is_sm120_supported() or is_sm90_supported()
        ), "NVFP4 KV cache quantize requires SM100/SM120 or SM90 fallback GPU"

        b, m, n = tensor.shape
        tensor_2d = tensor.reshape(b * m, n)

        # The KV cache path passes preloaded per-layer scales already on device.
        # Keep scalar/0-d support for tests and future fallback paths, but do not
        # silently move tensor scales here.
        if isinstance(global_scale, (int, float)):
            global_scale = torch.tensor(
                [global_scale], dtype=torch.float32, device=tensor.device
            )
        elif global_scale.dim() == 0:
            global_scale = global_scale.unsqueeze(0)
        elif global_scale.device != tensor.device:
            raise ValueError(
                "NVFP4 global scale tensor must already be on the KV tensor device."
            )

        if is_sm100_supported() or is_sm120_supported():
            from flashinfer import nvfp4_kv_quantize

            # nvfp4_kv_quantize takes global_scale directly (not inverted)
            fp4_2d, scales_2d = nvfp4_kv_quantize(tensor_2d, global_scale)
        else:
            # SM90: fp4_quantize takes inverted global_scale
            from flashinfer import fp4_quantize

            global_scale_inv = 1.0 / global_scale
            fp4_2d, scales_2d = fp4_quantize(
                tensor_2d,
                global_scale_inv,
                sf_vec_size=16,
                sf_use_ue8m0=False,

View on GitHub (pinned to 0132848349)

Solutions

1. Pass a plain float/int for global_scale and let the function construct the tensor on tensor.device
2. Or explicitly move the scale: global_scale = global_scale.to(tensor.device) before calling quantize
3. Re-create scale tensors after any device move instead of caching stale ones

Example fix

# before
scale = torch.tensor([2.0])  # CPU
quant_tensor = quantize(kv_tensor, scale)
# after
quant_tensor = quantize(kv_tensor, 2.0)  # or scale.to(kv_tensor.device)

if isinstance(global_scale, torch.Tensor):
    assert global_scale.device == tensor.device, "scale device mismatch"
    if global_scale.dim() == 0:
        global_scale = global_scale.unsqueeze(0)

When it happens

Trigger: Passing a torch tensor global_scale created on CPU (default device) while the hidden states / KV tensor lives on cuda:0; reusing a cached scale tensor after moving the model between devices.

Common situations: Integrating NVFP4 KV quantization in custom attention code; device migrations (model.to('cuda')) after scale tensors were captured; multi-GPU setups where scale was pinned to one rank's device.

Related errors

• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• {name} must be on q's device {device}, got {scale.device}
• kv-canary: {name} must be on {reference_name}'s device {refe
• Type must match: {self.a_dtype} != {self.b_dtype}
• nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a4b9c57fc81f064f. Report an issue: GitHub.