sgl-project/sglang · error · ValueError

NVFP4 global scale tensor must already be on the KV tensor…

Error message

NVFP4 global scale tensor must already be on the KV tensor device.

What it means

NVFP4 KV quantize accepts a global_scale that is a Python number, or a tensor already on the same device as the KV tensor. If a tensor is supplied and its device differs from tensor.device (e.g. scale still on CPU while KV states are on GPU), this ValueError is raised rather than silently triggering implicit cross-device copies in the quantization kernel path.

Solutions

  1. Pass a plain float/int for global_scale and let the function construct the tensor on tensor.device
  2. Or explicitly move the scale: global_scale = global_scale.to(tensor.device) before calling quantize
  3. Re-create scale tensors after any device move instead of caching stale ones

Example fix

# before
scale = torch.tensor([2.0])  # CPU
quant_tensor = quantize(kv_tensor, scale)
# after
quant_tensor = quantize(kv_tensor, 2.0)  # or scale.to(kv_tensor.device)
Defensive patterns

Strategy: validation

Validate before calling

if isinstance(global_scale, torch.Tensor):
    assert global_scale.device == tensor.device, "scale device mismatch"
    if global_scale.dim() == 0:
        global_scale = global_scale.unsqueeze(0)

Prevention

When it happens

Trigger: Passing a torch tensor global_scale created on CPU (default device) while the hidden states / KV tensor lives on cuda:0; reusing a cached scale tensor after moving the model between devices.

Common situations: Integrating NVFP4 KV quantization in custom attention code; device migrations (model.to('cuda')) after scale tensors were captured; multi-GPU setups where scale was pinned to one rank's device.

Related errors


AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a4b9c57fc81f064f. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/layers/quantization/kvfp4_tensor.py:201

        assert (
            is_sm100_supported() or is_sm120_supported() or is_sm90_supported()
        ), "NVFP4 KV cache quantize requires SM100/SM120 or SM90 fallback GPU"

        b, m, n = tensor.shape
        tensor_2d = tensor.reshape(b * m, n)

        # The KV cache path passes preloaded per-layer scales already on device.
        # Keep scalar/0-d support for tests and future fallback paths, but do not
        # silently move tensor scales here.
        if isinstance(global_scale, (int, float)):
            global_scale = torch.tensor(
                [global_scale], dtype=torch.float32, device=tensor.device
            )
        elif global_scale.dim() == 0:
            global_scale = global_scale.unsqueeze(0)
        elif global_scale.device != tensor.device:
            raise ValueError(
                "NVFP4 global scale tensor must already be on the KV tensor device."
            )

        if is_sm100_supported() or is_sm120_supported():
            from flashinfer import nvfp4_kv_quantize

            # nvfp4_kv_quantize takes global_scale directly (not inverted)
            fp4_2d, scales_2d = nvfp4_kv_quantize(tensor_2d, global_scale)
        else:
            # SM90: fp4_quantize takes inverted global_scale
            from flashinfer import fp4_quantize

            global_scale_inv = 1.0 / global_scale
            fp4_2d, scales_2d = fp4_quantize(
                tensor_2d,
                global_scale_inv,
                sf_vec_size=16,
                sf_use_ue8m0=False,

View on GitHub (pinned to 0132848349)