sgl-project/sglang · error · ValueError

MXFP8 KV cache requires v_head_dim divisible by {self.MXFP8_

Error message

MXFP8 KV cache requires v_head_dim divisible by {self.MXFP8_SCALE_BLOCK_SIZE}, got {v}.

What it means

Same block-size constraint as for k, but applied to v_head_dim: MXFP8 scale blocks along the value head dimension require v_head_dim % MXFP8_SCALE_BLOCK_SIZE == 0. Models with differing k/v head dims (e.g. v smaller than k) commonly violate this.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3365

    def _create_buffers(self):
        with self.memory_saver_adapter.region(GPU_MEMORY_TYPE_KV_CACHE):
            with (
                torch.cuda.use_mem_pool(self.custom_mem_pool)
                if self.enable_custom_mem_pool
                else nullcontext()
            ):
                m = self.size + self.page_size
                n = self.head_num
                k = self.head_dim
                v = self.v_head_dim

                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}."
                    )
                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires v_head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}."
                    )
                if not hasattr(torch, "float8_e8m0fnu"):
                    raise RuntimeError(
                        "MXFP8 KV cache requires torch.float8_e8m0fnu support."
                    )
                if self.use_hnd:
                    # Buffers are NHD; the inherited HND move_kv_cache branch
                    # would silently relocate wrong bytes.
                    raise ValueError(
                        "MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE."
                    )

                self.store_dtype = torch.float8_e4m3fn
                self.k_buffer = [
                    torch.zeros((m, n, k), dtype=self.store_dtype, device=self.device)
                    for _ in range(self.layer_num)

View on GitHub (pinned to 0132848349)

Solutions

Use a model whose v_head_dim is divisible by 32
Fall back to a non-block-scaled kv-cache-dtype (fp8_e4m3 or bf16)
Check model config (hidden_size/num_value_heads) before enabling MXFP8 KV cache

Example fix

# before
server_args = ServerArgs(kv_cache_dtype='mxfp8', ...)  # model with v_head_dim=80
# after
server_args = ServerArgs(kv_cache_dtype='fp8_e4m3', ...)  # no block-size constraint

Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.mem_cache.memory_pool import MHATokenToKVPoolMXFP8
bs = MHATokenToKVPoolMXFP8.MXFP8_SCALE_BLOCK_SIZE
assert config.v_head_dim % bs == 0 and config.head_dim % bs == 0

Try / catch

try:
    pool = MHATokenToKVPoolMXFP8(...)
except ValueError as e:
    if 'v_head_dim divisible' in str(e):
        use_dtype('fp8_e4m3')

Prevention

Check v_head_dim separately from head_dim in GQA/MQA models
Validate both dims against block size at config load time

When it happens

Trigger: MXFP8 KV cache enabled on a model whose v_head_dim is not a multiple of the scale block size (32), e.g. v_head_dim=80 or models using GQA with reduced v dimension.

Common situations: Multi-query / grouped-query models where v_head_dim differs from head_dim and is not block aligned; enabling MXFP8 KV cache on models not validated for it.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7fd0860e94ad31ad. Report an issue: GitHub.