{"record":{"id":"bd9bbd5228d9cf06","repo":"sgl-project/sglang","slug":"mxfp8-kv-cache-requires-torch-float8-e8m0fnu-suppo","errorCode":null,"errorMessage":"MXFP8 KV cache requires torch.float8_e8m0fnu support.","messagePattern":"MXFP8 KV cache requires torch\\.float8_e8m0fnu support\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/memory_pool.py","lineNumber":3370,"sourceCode":"                else nullcontext()\n            ):\n                m = self.size + self.page_size\n                n = self.head_num\n                k = self.head_dim\n                v = self.v_head_dim\n\n                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}.\"\n                    )\n                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires v_head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}.\"\n                    )\n                if not hasattr(torch, \"float8_e8m0fnu\"):\n                    raise RuntimeError(\n                        \"MXFP8 KV cache requires torch.float8_e8m0fnu support.\"\n                    )\n                if self.use_hnd:\n                    # Buffers are NHD; the inherited HND move_kv_cache branch\n                    # would silently relocate wrong bytes.\n                    raise ValueError(\n                        \"MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE.\"\n                    )\n\n                self.store_dtype = torch.float8_e4m3fn\n                self.k_buffer = [\n                    torch.zeros((m, n, k), dtype=self.store_dtype, device=self.device)\n                    for _ in range(self.layer_num)\n                ]\n                self.v_buffer = [\n                    torch.zeros((m, n, v), dtype=self.store_dtype, device=self.device)\n                    for _ in range(self.layer_num)\n                ]","sourceCodeStart":3352,"sourceCodeEnd":3388,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/memory_pool.py#L3352-L3388","documentation":"MXFP8 uses e8m0fnu (8-bit exponent-only power-of-two scales), a dtype introduced in recent PyTorch versions. If the installed torch lacks torch.float8_e8m0fnu, the MXFP8 pool cannot build its scale buffers and raises RuntimeError at init.","triggerScenarios":"Constructing MHATokenToKVPoolMXFP8 with an older PyTorch that does not define torch.float8_e8m0fnu (hasattr check fails).","commonSituations":"Running an older torch (<2.7-ish) in the environment while selecting the MXFP8 KV cache dtype; mismatched sglang/torch versions after an upgrade; conda/pip environment drift.","solutions":["Upgrade PyTorch to a version that includes torch.float8_e8m0fnu (>=2.7)","If upgrade is not possible, use a different kv-cache-dtype (fp8_e4m3 or bf16)","Verify with python -c \"import torch; print(hasattr(torch,'float8_e8m0fnu'))\" before launching"],"exampleFix":"# before\npip install torch==2.5.1  # lacks float8_e8m0fnu; MXFP8 KV cache init fails\n# after\npip install --upgrade torch  # >=2.7 provides torch.float8_e8m0fnu","handlingStrategy":"validation","validationCode":"import torch\nif not hasattr(torch, 'float8_e8m0fnu'):\n    raise SystemExit('MXFP8 KV cache needs torch>=2.7 with float8_e8m0fnu; upgrade or change kv_cache_dtype')","typeGuard":null,"tryCatchPattern":"try:\n    pool = MHATokenToKVPoolMXFP8(...)\nexcept RuntimeError as e:\n    if 'float8_e8m0fnu' in str(e):\n        use_dtype('fp8_e4m3')  # fallback","preventionTips":["Pin torch>=2.7 in requirements when MXFP8 is used","Run a capability probe script at deploy time","Add torch version to startup logs"],"tags":["mxfp8","kv-cache","pytorch-version","dtype"],"backgroundTag":"missing-dtype-support","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}