{"record":{"id":"4fc99ed2b75fbc04","repo":"sgl-project/sglang","slug":"mxfp8-kv-cache-requires-head-dim-divisible-by-sel","errorCode":null,"errorMessage":"MXFP8 KV cache requires head_dim divisible by {self.MXFP8_SCALE_BLOCK_SIZE}, got {k}.","messagePattern":"MXFP8 KV cache requires head_dim divisible by (.+?), got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/memory_pool.py","lineNumber":3360,"sourceCode":"    stored beside it and passed to the FA4 MXFP8 kernel.\n    \"\"\"\n\n    MXFP8_SCALE_BLOCK_SIZE = 32\n\n    def _create_buffers(self):\n        with self.memory_saver_adapter.region(GPU_MEMORY_TYPE_KV_CACHE):\n            with (\n                torch.cuda.use_mem_pool(self.custom_mem_pool)\n                if self.enable_custom_mem_pool\n                else nullcontext()\n            ):\n                m = self.size + self.page_size\n                n = self.head_num\n                k = self.head_dim\n                v = self.v_head_dim\n\n                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}.\"\n                    )\n                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires v_head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}.\"\n                    )\n                if not hasattr(torch, \"float8_e8m0fnu\"):\n                    raise RuntimeError(\n                        \"MXFP8 KV cache requires torch.float8_e8m0fnu support.\"\n                    )\n                if self.use_hnd:\n                    # Buffers are NHD; the inherited HND move_kv_cache branch\n                    # would silently relocate wrong bytes.\n                    raise ValueError(\n                        \"MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE.\"\n                    )","sourceCodeStart":3342,"sourceCodeEnd":3378,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/memory_pool.py#L3342-L3378","documentation":"MHATokenToKVPoolMXFP8 stores per-block MXFP8 scales along the head_dim axis with a fixed block size (MXFP8_SCALE_BLOCK_SIZE, typically 32). If the model's k head_dim is not divisible by that block size, scales cannot be laid out, so construction fails with a ValueError.","triggerScenarios":"Launching a model with --kv-cache-dtype fp8_e4m3/mxfp8 MXFP8 KV cache where head_dim (k) % MXFP8_SCALE_BLOCK_SIZE != 0, e.g. head_dim 80 or 96 with block size 32.","commonSituations":"Serving a model with an unusual head dimension (80, 12, 18 per-head dims) with MXFP8 KV cache enabled; switching from standard FP8 (row-wise scale) KV cache to MXFP8 block-scaled format.","solutions":["Pick a model whose head_dim is divisible by 32 (e.g. 64, 128, 256)","Use a different kv-cache-dtype (standard fp8_e4m3 or bfpp16/fp16) that does not require block-aligned head_dim","Pad/reshape is not supported; do not attempt to work around at runtime"],"exampleFix":"# before\nserver_args = ServerArgs(kv_cache_dtype='fp8_e4m3', ...)  # routed to MXFP8 pool, head_dim=80\n# after\nserver_args = ServerArgs(kv_cache_dtype='bf16', ...)  # or use a model with head_dim % 32 == 0","handlingStrategy":"validation","validationCode":"from sglang.srt.mem_cache.memory_pool import MHATokenToKVPoolMXFP8\nassert model_head_dim % MHATokenToKVPoolMXFP8.MXFP8_SCALE_BLOCK_SIZE == 0, 'head_dim not MXFP8-aligned'","typeGuard":null,"tryCatchPattern":"try:\n    pool = MHATokenToKVPoolMXFP8(...)\nexcept ValueError as e:\n    if 'divisible by' in str(e):\n        fall_back_to_standard_fp8_pool()","preventionTips":["Read head_dim from model config before enabling MXFP8 KV cache","Add a startup assertion for head_dim % 32 == 0","Maintain a list of MXFP8-validated models"],"tags":["kv-cache","mxfp8","quantization","head-dim"],"backgroundTag":"quantization-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}