{"record":{"id":"981881915e8b7ed6","repo":"sgl-project/sglang","slug":"mxfp8-kv-cache-does-not-support-sglang-use-hnd-kvc","errorCode":null,"errorMessage":"MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE.","messagePattern":"MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/memory_pool.py","lineNumber":3376,"sourceCode":"\n                if k % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {k}.\"\n                    )\n                if v % self.MXFP8_SCALE_BLOCK_SIZE != 0:\n                    raise ValueError(\n                        f\"MXFP8 KV cache requires v_head_dim divisible by \"\n                        f\"{self.MXFP8_SCALE_BLOCK_SIZE}, got {v}.\"\n                    )\n                if not hasattr(torch, \"float8_e8m0fnu\"):\n                    raise RuntimeError(\n                        \"MXFP8 KV cache requires torch.float8_e8m0fnu support.\"\n                    )\n                if self.use_hnd:\n                    # Buffers are NHD; the inherited HND move_kv_cache branch\n                    # would silently relocate wrong bytes.\n                    raise ValueError(\n                        \"MXFP8 KV cache does not support SGLANG_USE_HND_KVCACHE.\"\n                    )\n\n                self.store_dtype = torch.float8_e4m3fn\n                self.k_buffer = [\n                    torch.zeros((m, n, k), dtype=self.store_dtype, device=self.device)\n                    for _ in range(self.layer_num)\n                ]\n                self.v_buffer = [\n                    torch.zeros((m, n, v), dtype=self.store_dtype, device=self.device)\n                    for _ in range(self.layer_num)\n                ]\n\n                # UE8M0 scales, one per 32-element block. For the production\n                # page_size==128 path they are stored interleaved in the FA4\n                # BlockScaledBasicChunk atom layout\n                # (num_pages, head, 32, page_size//32, sf_dim) and written by\n                # the store_sf_interleaved kernel; otherwise flat per slot. Must","sourceCodeStart":3358,"sourceCodeEnd":3394,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/memory_pool.py#L3358-L3394","documentation":"MXFP8 pool buffers are allocated in NHD layout (token, head, dim) with separate scale tensors, so the inherited move_kv_cache HND branch would relocate wrong bytes. Setting SGLANG_USE_HND_KVCACHE=1 therefore raises ValueError at pool construction.","triggerScenarios":"Setting environment variable SGLANG_USE_HND_KVCACHE=1 (or otherwise defaulting use_hnd True) while the MXFP8 KV pool class is selected.","commonSituations":"Carrying over an HND (head-num-dim) KV cache env toggle from another setup or model when switching to MXFP8 KV cache; leftover env vars in launch scripts/containers.","solutions":["Unset SGLANG_USE_HND_KVCACHE (or set to 0) when using MXFP8 KV cache","Audit launch scripts/Dockerfiles/compose env for stale SGLANG_* toggles","If HND layout is required, use a non-MXFP8 kv-cache-dtype"],"exampleFix":"# before\nSGLANG_USE_HND_KVCACHE=1 python -m sglang.launch_server --kv-cache-dtype fp8_e4m3(MXFP8) ...\n# after\nunset SGLANG_USE_HND_KVCACHE; python -m sglang.launch_server --kv-cache-dtype fp8_e4m3 ...","handlingStrategy":"validation","validationCode":"import os\nassert not os.environ.get('SGLANG_USE_HND_KVCACHE', '').lower() in ('1','true'), 'HND layout incompatible with MXFP8 KV cache'","typeGuard":null,"tryCatchPattern":"try:\n    pool = MHATokenToKVPoolMXFP8(...)\nexcept ValueError as e:\n    if 'HND' in str(e):\n        os.environ.pop('SGLANG_USE_HND_KVCACHE', None); retry_launch()","preventionTips":["Scrub SGLANG_* env vars per workload","Encode MXFP8+HND incompatibility in server args validation","Document env var incompatibilities in the launch script"],"tags":["kv-cache","mxfp8","hnd-layout","env-var"],"backgroundTag":"conflicting-env-flags","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}