{"record":{"id":"bd48a56c41f80eb4","repo":"sgl-project/sglang","slug":"only-support-per-tensor-scaling-factor-for-fp8-kv","errorCode":null,"errorMessage":"Only support per-tensor scaling factor for fp8 KV cache","messagePattern":"Only support per-tensor scaling factor for fp8 KV cache","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/kv_cache.py","lineNumber":77,"sourceCode":"        elif layer.k_scale <= 0.0 and layer.v_scale <= 0.0:\n            # If no scales were loaded (both scales are invalid non-positive\n            # values), use the default value of 1.0\n            k_scale = 1.0\n            v_scale = 1.0\n        else:\n            # If we find a single kv_scale in the checkpoint, we remap\n            # kv_scale to k_scale during weight loading, and duplicate\n            # k_scale to v_scale here\n            assert layer.k_scale > 0.0\n            scale_to_duplicate = max(layer.k_scale, layer.v_scale)\n            k_scale = scale_to_duplicate.to(\"cpu\").tolist()\n            v_scale = scale_to_duplicate.to(\"cpu\").tolist()\n            if is_fp8_fnuz():\n                k_scale *= 2\n                v_scale *= 2\n\n        if not isinstance(k_scale, float) or not isinstance(v_scale, float):\n            raise ValueError(\n                \"Only support per-tensor scaling factor \" \"for fp8 KV cache\"\n            )\n\n        # These are used in the final Attention.forward()\n        layer.k_scale.copy_(k_scale)\n        layer.v_scale.copy_(v_scale)\n        layer.k_scale_float = k_scale\n        layer.v_scale_float = v_scale\n","sourceCodeStart":59,"sourceCodeEnd":86,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/kv_cache.py#L59-L86","documentation":"For FP8 KV caches, process_weights_after_loading requires per-tensor (scalar) k_scale/v_scale: after .to('cpu').tolist() both values must be Python floats. If the checkpoint ships per-channel/per-token scale tensors, tolist() returns a list and this ValueError fires, because the runtime only supports a single scalar scale per KV tensor.","triggerScenarios":"Loading a model whose k_scale/v_scale checkpoint tensors have shape [num_heads] or [num_kv_heads, head_dim] instead of a 0-d/scalar tensor; calibrated checkpoints exported with per-head scales.","commonSituations":"Using FP8 KV cache quantization (--kv-cache-dtype fp8_e4m3) with a checkpoint quantized by a tool that emits non-scalar scales; mixing calibration formats between vLLM/SGLang exports.","solutions":["Use a checkpoint with scalar k_scale/v_scale (or none at all, letting calibration derive one scalar)","Re-export/re-calibrate the KV scales to per-tensor format","Fall back to --kv-cache-dtype auto (BF16 KV cache) if per-channel scales are required"],"exampleFix":"# checkpoint (before): k_scale shape [32] (per-head)\n# checkpoint (after):  k_scale shape [] scalar tensor, e.g. tensor(1.7)","handlingStrategy":"validation","validationCode":"for name in (\"k_scale\", \"v_scale\"):\n    t = getattr(layer, name, None)\n    if t is not None and t.dim() != 0:\n        raise ValueError(f\"{name} must be scalar for FP8 KV cache, got shape {tuple(t.shape)}\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Verify checkpoint k_scale/v_scale are 0-d tensors before enabling --kv-cache-dtype fp8","Prefer checkpoints calibrated with per-tensor KV scales"],"tags":["kv-cache","fp8","scale-format","checkpoint-validation"],"backgroundTag":"unsupported-quantization-scale-format","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}