{"record":{"id":"a1c05befb00e289b","repo":"sgl-project/sglang","slug":"mxfp8-kv-cache-requires-k-and-v-scale-tensors","errorCode":null,"errorMessage":"MXFP8 KV cache requires K and V scale tensors.","messagePattern":"MXFP8 KV cache requires K and V scale tensors\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/memory_pool.py","lineNumber":3498,"sourceCode":"        dcp_kv_mask: Optional[torch.Tensor] = None,\n    ):\n        if dcp_kv_mask is not None:\n            raise NotImplementedError(\"MXFP8 KV cache does not support DCP KV masks.\")\n        loc, _, _ = unwrap_write_loc(loc_info)\n        maybe_detect_oob(\n            loc, 0, self.size + self.page_size, \"set_kv_buffer (MHA-MXFP8)\"\n        )\n        layer_id = (\n            layer_id_override if layer_id_override is not None else layer.layer_id\n        )\n        idx = layer_id - self.start_layer\n\n        if k_scale is None or v_scale is None:\n            # Fused path (SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE): the layer\n            # hands us bf16 K/V and one kernel quantizes + scatters the fp8\n            # payload and the interleaved UE8M0 scales.\n            if not self.mxfp8_sf_interleaved or cache_k.dtype == self.store_dtype:\n                raise ValueError(\"MXFP8 KV cache requires K and V scale tensors.\")\n            from sglang.kernels.ops.quantization.mxfp8_quant import quant_store_kv_mxfp8\n\n            quant_store_kv_mxfp8(\n                cache_k,\n                cache_v,\n                loc,\n                self.k_buffer[idx],\n                self.v_buffer[idx],\n                self.k_scale_buffer[idx],\n                self.v_scale_buffer[idx],\n                page_size=self.page_size,\n            )\n            return\n\n        from sglang.srt.model_executor.runner import get_is_capture_mode\n\n        if get_is_capture_mode() and self.alt_stream is not None:\n            current_stream = self.device_module.current_stream()","sourceCodeStart":3480,"sourceCodeEnd":3516,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/memory_pool.py#L3480-L3516","documentation":"MHAMXFP8TokenToKVPool.set_kv_buffer was called without k_scale/v_scale, and the fused quantize-and-store fallback path is unavailable. The fused path (SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE) only works when the pool uses interleaved scale-factor layout and the incoming cache_k is still bf16 (not already the fp8 store dtype). If either precondition fails, the pool has no way to produce the MXFP8 payload plus UE8M0 scales, so it raises.","triggerScenarios":"Calling set_kv_buffer on the MXFP8 KV pool with k_scale=None or v_scale=None while mxfp8_sf_interleaved is False, or while cache_k.dtype already equals self.store_dtype (already quantized fp8), i.e. the model layer did not pass scales and did not opt into the fused quant kernel.","commonSituations":"Enabling MXFP8 KV cache quantization on a model whose attention layers never populate layer.k_scale/layer.v_scale; disabling SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE; a model path that pre-quantizes K/V to fp8 before calling set_kv_buffer; version changes that renamed or stopped propagating the scale attributes.","solutions":["Ensure the attention layer exposes k_scale/v_scale (e.g. layer.k_scale is not None) so the scaled path is taken","Enable the fused path: set SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE=1 and confirm the pool was constructed with mxfp8_sf_interleaved=True","If you pre-quantize K/V yourself, pass the scales explicitly instead of relying on the fused kernel","If the model does not support MXFP8 KV cache, disable KV cache MXFP8 quantization (--kv-cache-dtype fp8_e4m3 or bf16)"],"exampleFix":"// before\npool.set_kv_buffer(layer, loc, cache_k, cache_v)  # k_scale/v_scale None, non-interleaved pool -> ValueError\n// after\npool.set_kv_buffer(layer, loc, cache_k, cache_v, k_scale=layer.k_scale, v_scale=layer.v_scale)\n// or: export SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE=1 with interleaved SF layout","handlingStrategy":"validation","validationCode":"assert layer.k_scale is not None and layer.v_scale is not None, 'MXFP8 KV cache needs k_scale/v_scale'\nassert pool.mxfp8_sf_interleaved and cache_k.dtype != pool.store_dtype or (layer.k_scale is not None), 'no fused path and no scales'","typeGuard":"def has_mxfp8_scales(layer) -> bool:\n    return layer.k_scale is not None and layer.v_scale is not None","tryCatchPattern":"try:\n    pool.set_kv_buffer(layer, loc, cache_k, cache_v, k_scale=layer.k_scale, v_scale=layer.v_scale)\nexcept ValueError as e:\n    if 'MXFP8' in str(e):\n        raise RuntimeError('Model does not provide KV scales; disable MXFP8 kv-cache-dtype') from e\n    raise","preventionTips":["Always pass k_scale/v_scale explicitly when kv-cache-dtype is mxfp8","Gate MXFP8 KV cache on model support (layer.k_scale present)","Test server startup with MXFP8 KV cache in CI for supported models only"],"tags":["mxfp8","kv-cache","quantization","scale-tensors","sglang"],"backgroundTag":"missing-quantization-scale","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}