sgl-project/sglang · error · NotImplementedError

prefix-valid commit is unsupported for MXFP8 KV cache (it do

Error message

prefix-valid commit is unsupported for MXFP8 KV cache (it does not carry the scale buffers).

What it means

MHAMXFP8TokenToKVPool deliberately does not implement set_kv_buffer_prefix_valid, the prefix-valid commit path that copies raw KV buffers. Because the MXFP8 pool stores fp8 payloads with separate scale buffers, a raw-copy commit would drop the scales and corrupt the cache, so the operation is refused.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3680

    def get_kv_scale_buf_infos(self):
        """(ptrs, lens, item_lens) for the UE8M0 scale buffers, k then v.

        The interleaved layout puts pages on the leading axis, so a page's
        scales are one contiguous row; the flat layout is per slot.
        """
        tensors = self.k_scale_buffer + self.v_scale_buffer
        ptrs = [t.data_ptr() for t in tensors]
        lens = [t.nbytes for t in tensors]
        row_bytes = [t[0].nbytes for t in tensors]
        if self.mxfp8_sf_interleaved:
            item_lens = row_bytes
        else:
            item_lens = [rb * self.page_size for rb in row_bytes]
        return ptrs, lens, item_lens

    def set_kv_buffer_prefix_valid(self, *args, **kwargs):
        raise NotImplementedError(
            "prefix-valid commit is unsupported for MXFP8 KV cache "
            "(it does not carry the scale buffers)."
        )

    def get_kv_size_bytes(self):
        k_size_bytes = 0
        v_size_bytes = 0
        for k_cache in self.k_buffer:
            k_size_bytes += get_tensor_size_bytes(k_cache)
        for k_scale in self.k_scale_buffer:
            k_size_bytes += get_tensor_size_bytes(k_scale)
        for v_cache in self.v_buffer:
            v_size_bytes += get_tensor_size_bytes(v_cache)
        for v_scale in self.v_scale_buffer:
            v_size_bytes += get_tensor_size_bytes(v_scale)
        return k_size_bytes, v_size_bytes

View on GitHub (pinned to 0132848349)

Solutions

1. Disable the feature that uses prefix-valid commits (e.g. turn off the specific prefix-transfer option or hierarchical cache mode) when using MXFP8 KV cache
2. Use a non-MXFP8 kv-cache-dtype if you need prefix-valid commits
3. Wait for / implement scale-aware prefix commit in MHAMXFP8TokenToKVPool instead of the raw copy

Example fix

// before
--kv-cache-dtype mxfp8 --enable-hierarchical-cache  # triggers set_kv_buffer_prefix_valid
// after
--kv-cache-dtype fp8_e4m3 --enable-hierarchical-cache

if isinstance(pool, MHAMXFP8TokenToKVPool):
    raise SystemExit('prefix-valid commit unsupported with MXFP8 KV cache; disable it or change kv-cache-dtype')

def supports_prefix_valid(pool) -> bool:
    return not type(pool).__name__.startswith('MHAMXFP8')

try:
    pool.set_kv_buffer_prefix_valid(*args)
except NotImplementedError as e:
    logger.warning('skipping prefix-valid commit: %s', e)

When it happens

Trigger: Any code path that commits KV entries via set_kv_buffer_prefix_valid (prefix caching reuse / transfer flows that mark prefixes valid by pointer copy) while the memory pool is an MXFP8 KV pool.

Common situations: Enabling prefix caching or a KV-transfer/HiCache feature that uses prefix-valid commits together with MXFP8 KV cache quantization; newer SGLang versions adding new prefix-commit codepaths that reach this pool.

Related errors

• MXFP8 KV cache requires the FA4 backend.
• MXFP8 KV cache does not support DCP KV masks.
• MXFP8 KV cache requires K and V scale tensors.
• move_kv_cache is not yet supported for MiniMaxSparseKVPool:
• get_split_heads_page_buffer_meta requires layout='page_head'

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/64b40021e9b9be6e. Report an issue: GitHub.