sgl-project/sglang · error · ValueError

MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE.

Error message

MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE.

What it means

MXFP8 pool buffers are allocated in NHD layout (token, head, dim) with separate scale tensors, so the inherited move_kv_cache HND branch would relocate wrong bytes. Setting SGLANG_USE_HND_KVCACHE=1 therefore raises ValueError at pool construction.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3376

                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}."
                    )
                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires v_head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}."
                    )
                if not hasattr(torch, "float8_e8m0fnu"):
                    raise RuntimeError(
                        "MXFP8 KV cache requires torch.float8_e8m0fnu support."
                    )
                if self.use_hnd:
                    # Buffers are NHD; the inherited HND move_kv_cache branch
                    # would silently relocate wrong bytes.
                    raise ValueError(
                        "MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE."
                    )

                self.store_dtype = torch.float8_e4m3fn
                self.k_buffer = [
                    torch.zeros((m, n, k), dtype=self.store_dtype, device=self.device)
                    for _ in range(self.layer_num)
                ]
                self.v_buffer = [
                    torch.zeros((m, n, v), dtype=self.store_dtype, device=self.device)
                    for _ in range(self.layer_num)
                ]

                # UE8M0 scales, one per 32-element block. For the production
                # page_size==128 path they are stored interleaved in the FA4
                # BlockScaledBasicChunk atom layout
                # (num_pages, head, 32, page_size//32, sf_dim) and written by
                # the store_sf_interleaved kernel; otherwise flat per slot. Must

View on GitHub (pinned to 0132848349)

Solutions

1. Unset SGLANG_USE_HND_KVCACHE (or set to 0) when using MXFP8 KV cache
2. Audit launch scripts/Dockerfiles/compose env for stale SGLANG_* toggles
3. If HND layout is required, use a non-MXFP8 kv-cache-dtype

Example fix

# before
SGLANG_USE_HND_KVCACHE=1 python -m sglang.launch_server --kv-cache-dtype fp8_e4m3(MXFP8) ...
# after
unset SGLANG_USE_HND_KVCACHE; python -m sglang.launch_server --kv-cache-dtype fp8_e4m3 ...

import os
assert not os.environ.get('SGLANG_USE_HND_KVCACHE', '').lower() in ('1','true'), 'HND layout incompatible with MXFP8 KV cache'

try:
    pool = MHATokenToKVPoolMXFP8(...)
except ValueError as e:
    if 'HND' in str(e):
        os.environ.pop('SGLANG_USE_HND_KVCACHE', None); retry_launch()

When it happens

Trigger: Setting environment variable SGLANG_USE_HND_KVCACHE=1 (or otherwise defaulting use_hnd True) while the MXFP8 KV pool class is selected.

Common situations: Carrying over an HND (head-num-dim) KV cache env toggle from another setup or model when switching to MXFP8 KV cache; leftover env vars in launch scripts/containers.

Related errors

• MXFP8 fused prologue requires interleaved K/V scale buffers
• MXFP8 fused prologue requires contiguous interleaved SFK/SFV
• MXFP8 fused decode prologue requires K/V scale buffers.
• MXFP8 KV cache requires the FA4 backend.
• MXFP8 KV cache requires head_dim divisible by {self.MXFP8_SC

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/981881915e8b7ed6. Report an issue: GitHub.