sgl-project/sglang · critical · ValueError

MXFP8 KV cache requires K and V scale tensors.

Error message

MXFP8 KV cache requires K and V scale tensors.

What it means

MHAMXFP8TokenToKVPool.set_kv_buffer was called without k_scale/v_scale, and the fused quantize-and-store fallback path is unavailable. The fused path (SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE) only works when the pool uses interleaved scale-factor layout and the incoming cache_k is still bf16 (not already the fp8 store dtype). If either precondition fails, the pool has no way to produce the MXFP8 payload plus UE8M0 scales, so it raises.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3498

        dcp_kv_mask: Optional[torch.Tensor] = None,
    ):
        if dcp_kv_mask is not None:
            raise NotImplementedError("MXFP8 KV cache does not support DCP KV masks.")
        loc, _, _ = unwrap_write_loc(loc_info)
        maybe_detect_oob(
            loc, 0, self.size + self.page_size, "set_kv_buffer (MHA-MXFP8)"
        )
        layer_id = (
            layer_id_override if layer_id_override is not None else layer.layer_id
        )
        idx = layer_id - self.start_layer

        if k_scale is None or v_scale is None:
            # Fused path (SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE): the layer
            # hands us bf16 K/V and one kernel quantizes + scatters the fp8
            # payload and the interleaved UE8M0 scales.
            if not self.mxfp8_sf_interleaved or cache_k.dtype == self.store_dtype:
                raise ValueError("MXFP8 KV cache requires K and V scale tensors.")
            from sglang.kernels.ops.quantization.mxfp8_quant import quant_store_kv_mxfp8

            quant_store_kv_mxfp8(
                cache_k,
                cache_v,
                loc,
                self.k_buffer[idx],
                self.v_buffer[idx],
                self.k_scale_buffer[idx],
                self.v_scale_buffer[idx],
                page_size=self.page_size,
            )
            return

        from sglang.srt.model_executor.runner import get_is_capture_mode

        if get_is_capture_mode() and self.alt_stream is not None:
            current_stream = self.device_module.current_stream()

View on GitHub (pinned to 0132848349)

Solutions

  1. Ensure the attention layer exposes k_scale/v_scale (e.g. layer.k_scale is not None) so the scaled path is taken
  2. Enable the fused path: set SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE=1 and confirm the pool was constructed with mxfp8_sf_interleaved=True
  3. If you pre-quantize K/V yourself, pass the scales explicitly instead of relying on the fused kernel
  4. If the model does not support MXFP8 KV cache, disable KV cache MXFP8 quantization (--kv-cache-dtype fp8_e4m3 or bf16)

Example fix

// before
pool.set_kv_buffer(layer, loc, cache_k, cache_v)  # k_scale/v_scale None, non-interleaved pool -> ValueError
// after
pool.set_kv_buffer(layer, loc, cache_k, cache_v, k_scale=layer.k_scale, v_scale=layer.v_scale)
// or: export SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE=1 with interleaved SF layout
Defensive patterns

Strategy: validation

Validate before calling

assert layer.k_scale is not None and layer.v_scale is not None, 'MXFP8 KV cache needs k_scale/v_scale'
assert pool.mxfp8_sf_interleaved and cache_k.dtype != pool.store_dtype or (layer.k_scale is not None), 'no fused path and no scales'

Type guard

def has_mxfp8_scales(layer) -> bool:
    return layer.k_scale is not None and layer.v_scale is not None

Try / catch

try:
    pool.set_kv_buffer(layer, loc, cache_k, cache_v, k_scale=layer.k_scale, v_scale=layer.v_scale)
except ValueError as e:
    if 'MXFP8' in str(e):
        raise RuntimeError('Model does not provide KV scales; disable MXFP8 kv-cache-dtype') from e
    raise

Prevention

When it happens

Trigger: Calling set_kv_buffer on the MXFP8 KV pool with k_scale=None or v_scale=None while mxfp8_sf_interleaved is False, or while cache_k.dtype already equals self.store_dtype (already quantized fp8), i.e. the model layer did not pass scales and did not opt into the fused quant kernel.

Common situations: Enabling MXFP8 KV cache quantization on a model whose attention layers never populate layer.k_scale/layer.v_scale; disabling SGLANG_OPT_INKLING_MXFP8_FUSED_QUANT_STORE; a model path that pre-quantizes K/V to fp8 before calling set_kv_buffer; version changes that renamed or stopped propagating the scale attributes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a1c05befb00e289b. Report an issue: GitHub.