sgl-project/sglang · error · ValueError

Only support per-tensor scaling factor for fp8 KV cache

Error message

Only support per-tensor scaling factor for fp8 KV cache

What it means

For FP8 KV caches, process_weights_after_loading requires per-tensor (scalar) k_scale/v_scale: after .to('cpu').tolist() both values must be Python floats. If the checkpoint ships per-channel/per-token scale tensors, tolist() returns a list and this ValueError fires, because the runtime only supports a single scalar scale per KV tensor.

Source

Thrown at python/sglang/srt/layers/quantization/kv_cache.py:77

        elif layer.k_scale <= 0.0 and layer.v_scale <= 0.0:
            # If no scales were loaded (both scales are invalid non-positive
            # values), use the default value of 1.0
            k_scale = 1.0
            v_scale = 1.0
        else:
            # If we find a single kv_scale in the checkpoint, we remap
            # kv_scale to k_scale during weight loading, and duplicate
            # k_scale to v_scale here
            assert layer.k_scale > 0.0
            scale_to_duplicate = max(layer.k_scale, layer.v_scale)
            k_scale = scale_to_duplicate.to("cpu").tolist()
            v_scale = scale_to_duplicate.to("cpu").tolist()
            if is_fp8_fnuz():
                k_scale *= 2
                v_scale *= 2

        if not isinstance(k_scale, float) or not isinstance(v_scale, float):
            raise ValueError(
                "Only support per-tensor scaling factor " "for fp8 KV cache"
            )

        # These are used in the final Attention.forward()
        layer.k_scale.copy_(k_scale)
        layer.v_scale.copy_(v_scale)
        layer.k_scale_float = k_scale
        layer.v_scale_float = v_scale

View on GitHub (pinned to 0132848349)

Solutions

1. Use a checkpoint with scalar k_scale/v_scale (or none at all, letting calibration derive one scalar)
2. Re-export/re-calibrate the KV scales to per-tensor format
3. Fall back to --kv-cache-dtype auto (BF16 KV cache) if per-channel scales are required

Example fix

# checkpoint (before): k_scale shape [32] (per-head)
# checkpoint (after):  k_scale shape [] scalar tensor, e.g. tensor(1.7)

for name in ("k_scale", "v_scale"):
    t = getattr(layer, name, None)
    if t is not None and t.dim() != 0:
        raise ValueError(f"{name} must be scalar for FP8 KV cache, got shape {tuple(t.shape)}")

When it happens

Trigger: Loading a model whose k_scale/v_scale checkpoint tensors have shape [num_heads] or [num_kv_heads, head_dim] instead of a 0-d/scalar tensor; calibrated checkpoints exported with per-head scales.

Common situations: Using FP8 KV cache quantization (--kv-cache-dtype fp8_e4m3) with a checkpoint quantized by a tool that emits non-scalar scales; mixing calibration formats between vLLM/SGLang exports.

Related errors

• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• kv_scales must be fp32, got {kv_scales.dtype}
• D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE
• kv_scales shape {tuple(kv_scales.shape)} does not match expe
• kv must be torch.float8_e4m3fn, got {kv.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bd48a56c41f80eb4. Report an issue: GitHub.