{"record":{"id":"7fd0860e94ad31ad","repo":"sgl-project/sglang","slug":"mxfp8-kv-cache-requires-v-head-dim-divisible-by-s","errorCode":null,"errorMessage":"MXFP8 KV cache requires v_head_dim divisible by {self.MXFP8_SCALE_BLOCK_SIZE}, got {v}.","messagePattern":"MXFP8 KV cache requires v_head_dim divisible by (.+?), got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/memory_pool.py","lineNumber":3365,"sourceCode":"    def _create_buffers(self):\n        with self.memory_saver_adapter.region(GPU_MEMORY_TYPE_KV_CACHE):\n            with (\n                torch.cuda.use_mem_pool(self.custom_mem_pool)\n                if self.enable_custom_mem_pool\n                else nullcontext()\n            ):\n                m = self.size + self.page_size\n                n = self.head_num\n                k = self.head_dim\n                v = self.v_head_dim\n\n                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}.\"\n                    )\n                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires v_head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}.\"\n                    )\n                if not hasattr(torch, \"float8_e8m0fnu\"):\n                    raise RuntimeError(\n                        \"MXFP8 KV cache requires torch.float8_e8m0fnu support.\"\n                    )\n                if self.use_hnd:\n                    # Buffers are NHD; the inherited HND move_kv_cache branch\n                    # would silently relocate wrong bytes.\n                    raise ValueError(\n                        \"MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE.\"\n                    )\n\n                self.store_dtype = torch.float8_e4m3fn\n                self.k_buffer = [\n                    torch.zeros((m, n, k), dtype=self.store_dtype, device=self.device)\n                    for _ in range(self.layer_num)","sourceCodeStart":3347,"sourceCodeEnd":3383,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/memory_pool.py#L3347-L3383","documentation":"Same block-size constraint as for k, but applied to v_head_dim: MXFP8 scale blocks along the value head dimension require v_head_dim % MXFP8_SCALE_BLOCK_SIZE == 0. Models with differing k/v head dims (e.g. v smaller than k) commonly violate this.","triggerScenarios":"MXFP8 KV cache enabled on a model whose v_head_dim is not a multiple of the scale block size (32), e.g. v_head_dim=80 or models using GQA with reduced v dimension.","commonSituations":"Multi-query / grouped-query models where v_head_dim differs from head_dim and is not block aligned; enabling MXFP8 KV cache on models not validated for it.","solutions":["Use a model whose v_head_dim is divisible by 32","Fall back to a non-block-scaled kv-cache-dtype (fp8_e4m3 or bf16)","Check model config (hidden_size/num_value_heads) before enabling MXFP8 KV cache"],"exampleFix":"# before\nserver_args = ServerArgs(kv_cache_dtype='mxfp8', ...)  # model with v_head_dim=80\n# after\nserver_args = ServerArgs(kv_cache_dtype='fp8_e4m3', ...)  # no block-size constraint","handlingStrategy":"validation","validationCode":"from sglang.srt.mem_cache.memory_pool import MHATokenToKVPoolMXFP8\nbs = MHATokenToKVPoolMXFP8.MXFP8_SCALE_BLOCK_SIZE\nassert config.v_head_dim % bs == 0 and config.head_dim % bs == 0","typeGuard":null,"tryCatchPattern":"try:\n    pool = MHATokenToKVPoolMXFP8(...)\nexcept ValueError as e:\n    if 'v_head_dim divisible' in str(e):\n        use_dtype('fp8_e4m3')","preventionTips":["Check v_head_dim separately from head_dim in GQA/MQA models","Validate both dims against block size at config load time"],"tags":["kv-cache","mxfp8","quantization","v-head-dim"],"backgroundTag":"quantization-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}