sgl-project/sglang · error · ValueError

MXFP8 KV cache requires head_dim divisible by {self.MXFP8_SC

Error message

MXFP8 KV cache requires head_dim divisible by {self.MXFP8_SCALE_BLOCK_SIZE}, got {k}.

What it means

MHATokenToKVPoolMXFP8 stores per-block MXFP8 scales along the head_dim axis with a fixed block size (MXFP8_SCALE_BLOCK_SIZE, typically 32). If the model's k head_dim is not divisible by that block size, scales cannot be laid out, so construction fails with a ValueError.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3360

    stored beside it and passed to the FA4 MXFP8 kernel.
    """

    MXFP8_SCALE_BLOCK_SIZE = 32

    def _create_buffers(self):
        with self.memory_saver_adapter.region(GPU_MEMORY_TYPE_KV_CACHE):
            with (
                torch.cuda.use_mem_pool(self.custom_mem_pool)
                if self.enable_custom_mem_pool
                else nullcontext()
            ):
                m = self.size + self.page_size
                n = self.head_num
                k = self.head_dim
                v = self.v_head_dim

                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}."
                    )
                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires v_head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}."
                    )
                if not hasattr(torch, "float8_e8m0fnu"):
                    raise RuntimeError(
                        "MXFP8 KV cache requires torch.float8_e8m0fnu support."
                    )
                if self.use_hnd:
                    # Buffers are NHD; the inherited HND move_kv_cache branch
                    # would silently relocate wrong bytes.
                    raise ValueError(
                        "MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE."
                    )

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a model whose head_dim is divisible by 32 (e.g. 64, 128, 256)
2. Use a different kv-cache-dtype (standard fp8_e4m3 or bfpp16/fp16) that does not require block-aligned head_dim
3. Pad/reshape is not supported; do not attempt to work around at runtime

Example fix

# before
server_args = ServerArgs(kv_cache_dtype='fp8_e4m3', ...)  # routed to MXFP8 pool, head_dim=80
# after
server_args = ServerArgs(kv_cache_dtype='bf16', ...)  # or use a model with head_dim % 32 == 0

from sglang.srt.mem_cache.memory_pool import MHATokenToKVPoolMXFP8
assert model_head_dim % MHATokenToKVPoolMXFP8.MXFP8_SCALE_BLOCK_SIZE == 0, 'head_dim not MXFP8-aligned'

try:
    pool = MHATokenToKVPoolMXFP8(...)
except ValueError as e:
    if 'divisible by' in str(e):
        fall_back_to_standard_fp8_pool()

When it happens

Trigger: Launching a model with --kv-cache-dtype fp8_e4m3/mxfp8 MXFP8 KV cache where head_dim (k) % MXFP8_SCALE_BLOCK_SIZE != 0, e.g. head_dim 80 or 96 with block size 32.

Common situations: Serving a model with an unusual head dimension (80, 12, 18 per-head dims) with MXFP8 KV cache enabled; switching from standard FP8 (row-wise scale) KV cache to MXFP8 block-scaled format.

Related errors

• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.
• MXFP8 KV cache requires the FA4 backend.
• MXFP8 KV cache requires v_head_dim divisible by {self.MXFP8_
• MXFP8 KV cache requires K and V scale tensors.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4fc99ed2b75fbc04. Report an issue: GitHub.