sgl-project/sglang · error · RuntimeError

MXFP8 KV cache requires torch.float8_e8m0fnu support.

Error message

MXFP8 KV cache requires torch.float8_e8m0fnu support.

What it means

MXFP8 uses e8m0fnu (8-bit exponent-only power-of-two scales), a dtype introduced in recent PyTorch versions. If the installed torch lacks torch.float8_e8m0fnu, the MXFP8 pool cannot build its scale buffers and raises RuntimeError at init.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3370

                else nullcontext()
            ):
                m = self.size + self.page_size
                n = self.head_num
                k = self.head_dim
                v = self.v_head_dim

                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}."
                    )
                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:
                    raise ValueError(
                        f"MXFP8 KV cache requires v_head_dim divisible by "
                        f"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}."
                    )
                if not hasattr(torch, "float8_e8m0fnu"):
                    raise RuntimeError(
                        "MXFP8 KV cache requires torch.float8_e8m0fnu support."
                    )
                if self.use_hnd:
                    # Buffers are NHD; the inherited HND move_kv_cache branch
                    # would silently relocate wrong bytes.
                    raise ValueError(
                        "MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE."
                    )

                self.store_dtype = torch.float8_e4m3fn
                self.k_buffer = [
                    torch.zeros((m, n, k), dtype=self.store_dtype, device=self.device)
                    for _ in range(self.layer_num)
                ]
                self.v_buffer = [
                    torch.zeros((m, n, v), dtype=self.store_dtype, device=self.device)
                    for _ in range(self.layer_num)
                ]

View on GitHub (pinned to 0132848349)

Solutions

Upgrade PyTorch to a version that includes torch.float8_e8m0fnu (>=2.7)
If upgrade is not possible, use a different kv-cache-dtype (fp8_e4m3 or bf16)
Verify with python -c "import torch; print(hasattr(torch,'float8_e8m0fnu'))" before launching

Example fix

# before
pip install torch==2.5.1  # lacks float8_e8m0fnu; MXFP8 KV cache init fails
# after
pip install --upgrade torch  # >=2.7 provides torch.float8_e8m0fnu

Defensive patterns

Strategy: validation

Validate before calling

import torch
if not hasattr(torch, 'float8_e8m0fnu'):
    raise SystemExit('MXFP8 KV cache needs torch>=2.7 with float8_e8m0fnu; upgrade or change kv_cache_dtype')

Try / catch

try:
    pool = MHATokenToKVPoolMXFP8(...)
except RuntimeError as e:
    if 'float8_e8m0fnu' in str(e):
        use_dtype('fp8_e4m3')  # fallback

Prevention

Pin torch>=2.7 in requirements when MXFP8 is used
Run a capability probe script at deploy time
Add torch version to startup logs

When it happens

Trigger: Constructing MHATokenToKVPoolMXFP8 with an older PyTorch that does not define torch.float8_e8m0fnu (hasattr check fails).

Common situations: Running an older torch (<2.7-ish) in the environment while selecting the MXFP8 KV cache dtype; mismatched sglang/torch versions after an upgrade; conda/pip environment drift.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bd9bbd5228d9cf06. Report an issue: GitHub.