sgl-project/sglang · error · ValueError

Unsupported KV cache type for decode offload

Error message

Unsupported KV cache type for decode offload

What it means

Decode KV offload manager init requires the allocator's KV pool to be MHATokenToKVPool or MLATokenToKVPool; any other pool class (hybrid SWA pools, custom backends, new pool types) is rejected with ValueError since the host offload pool builder only knows these two layouts.

Source

Thrown at python/sglang/srt/disaggregation/decode_kvcache_offload_manager.py:61

        token_to_kv_pool_allocator: BaseTokenToKVPoolAllocator,
        tp_group: torch.distributed.ProcessGroup,
        tree_cache: BasePrefixCache,
    ) -> None:
        self.req_to_token_pool = req_to_token_pool
        self.token_to_kv_pool_allocator = token_to_kv_pool_allocator
        self.page_size = get_schedule().page_size
        self.request_counter = 0
        self.tree_cache = tree_cache
        env_stride = envs.SGLANG_HICACHE_DECODE_OFFLOAD_STRIDE.get()
        if env_stride is None or env_stride <= 0:
            self.offload_stride = self.page_size
        else:
            self.offload_stride = max(
                self.page_size, (env_stride // self.page_size) * self.page_size
            )
        kv_cache = self.token_to_kv_pool_allocator.get_kvcache()
        if not isinstance(kv_cache, (MHATokenToKVPool, MLATokenToKVPool)):
            raise ValueError("Unsupported KV cache type for decode offload")
        self.decode_host_mem_pool = build_kv_host_pool(
            kv_pool=kv_cache,
            page_size=self.page_size,
            use_mla=isinstance(kv_cache, MLATokenToKVPool),
        )

        self.tp_group = tp_group
        self.tp_world_size = torch.distributed.get_world_size(group=self.tp_group)

        hicache_storage_backend_extra_config = {}
        if get_memory().hicache_storage_backend_extra_config:
            try:
                hicache_storage_backend_extra_config = json.loads(
                    get_memory().hicache_storage_backend_extra_config
                )
            except json.JSONDecodeError as e:
                raise ValueError(
                    f"Invalid hicache storage backend extra config JSON: {e}"

View on GitHub (pinned to 0132848349)

Solutions

1. Use decode offload only with models whose pool is MHA or MLA (non-hybrid attention)
2. Disable decode KV offload (--disable-hierarchical-cache or equivalent) for hybrid/SWA models
3. Check upstream for hybrid-pool offload support and upgrade if added

from sglang.srt.mem_cache import MHATokenToKVPool, MLATokenToKVPool
pool = allocator.get_kvcache()
assert isinstance(pool, (MHATokenToKVPool, MLATokenToKVPool)), 'offload unsupported for this pool'

def offload_supported(allocator) -> bool:
    return isinstance(allocator.get_kvcache(), (MHATokenToKVPool, MLATokenToKVPool))

When it happens

Trigger: Enabling decode KV offload (hierarchical cache on the decode server) with a KV pool that is not plain MHA or MLA — e.g. a hybrid/SWA allocator used by sliding-window or hybrid-attention models.

Common situations: Enabling --enable-hierarchical-cache / decode offload on a hybrid-attention model; a new pool type introduced upstream that the offload manager hasn't learned yet.

Related errors

• Ascend PD transfer does not support HiSparse destination dev
• NIXL PD transfer does not support HiSparse combined with dec
• NIXL KV transfer has no KV memory segments
• NIXL heterogeneous-TP direct-to-host KV transfer is not impl
• The argument disaggregation-decode-enable-offload-kvcache is

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/78dc0f9186ad7526. Report an issue: GitHub.