{"record":{"id":"78dc0f9186ad7526","repo":"sgl-project/sglang","slug":"unsupported-kv-cache-type-for-decode-offload","errorCode":null,"errorMessage":"Unsupported KV cache type for decode offload","messagePattern":"Unsupported KV cache type for decode offload","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/decode_kvcache_offload_manager.py","lineNumber":61,"sourceCode":"        token_to_kv_pool_allocator: BaseTokenToKVPoolAllocator,\n        tp_group: torch.distributed.ProcessGroup,\n        tree_cache: BasePrefixCache,\n    ) -> None:\n        self.req_to_token_pool = req_to_token_pool\n        self.token_to_kv_pool_allocator = token_to_kv_pool_allocator\n        self.page_size = get_schedule().page_size\n        self.request_counter = 0\n        self.tree_cache = tree_cache\n        env_stride = envs.SGLANG_HICACHE_DECODE_OFFLOAD_STRIDE.get()\n        if env_stride is None or env_stride <= 0:\n            self.offload_stride = self.page_size\n        else:\n            self.offload_stride = max(\n                self.page_size, (env_stride // self.page_size) * self.page_size\n            )\n        kv_cache = self.token_to_kv_pool_allocator.get_kvcache()\n        if not isinstance(kv_cache, (MHATokenToKVPool, MLATokenToKVPool)):\n            raise ValueError(\"Unsupported KV cache type for decode offload\")\n        self.decode_host_mem_pool = build_kv_host_pool(\n            kv_pool=kv_cache,\n            page_size=self.page_size,\n            use_mla=isinstance(kv_cache, MLATokenToKVPool),\n        )\n\n        self.tp_group = tp_group\n        self.tp_world_size = torch.distributed.get_world_size(group=self.tp_group)\n\n        hicache_storage_backend_extra_config = {}\n        if get_memory().hicache_storage_backend_extra_config:\n            try:\n                hicache_storage_backend_extra_config = json.loads(\n                    get_memory().hicache_storage_backend_extra_config\n                )\n            except json.JSONDecodeError as e:\n                raise ValueError(\n                    f\"Invalid hicache storage backend extra config JSON: {e}\"","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/decode_kvcache_offload_manager.py#L43-L79","documentation":"Decode KV offload manager init requires the allocator's KV pool to be MHATokenToKVPool or MLATokenToKVPool; any other pool class (hybrid SWA pools, custom backends, new pool types) is rejected with ValueError since the host offload pool builder only knows these two layouts.","triggerScenarios":"Enabling decode KV offload (hierarchical cache on the decode server) with a KV pool that is not plain MHA or MLA — e.g. a hybrid/SWA allocator used by sliding-window or hybrid-attention models.","commonSituations":"Enabling --enable-hierarchical-cache / decode offload on a hybrid-attention model; a new pool type introduced upstream that the offload manager hasn't learned yet.","solutions":["Use decode offload only with models whose pool is MHA or MLA (non-hybrid attention)","Disable decode KV offload (--disable-hierarchical-cache or equivalent) for hybrid/SWA models","Check upstream for hybrid-pool offload support and upgrade if added"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from sglang.srt.mem_cache import MHATokenToKVPool, MLATokenToKVPool\npool = allocator.get_kvcache()\nassert isinstance(pool, (MHATokenToKVPool, MLATokenToKVPool)), 'offload unsupported for this pool'","typeGuard":"def offload_supported(allocator) -> bool:\n    return isinstance(allocator.get_kvcache(), (MHATokenToKVPool, MLATokenToKVPool))","tryCatchPattern":null,"preventionTips":["Check model attention type before enabling hierarchical cache on decode"],"tags":["disaggregation","kv-offload","hicache","unsupported-pool-type"],"backgroundTag":"unsupported-hardware-feature-combo","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}