sgl-project/sglang · error · ValueError

Speculative decoding with --enable-unified-memory is only su

Error message

Speculative decoding with --enable-unified-memory is only supported for hybrid-Mamba targets; the unified hybrid-SWA pool's draft sizing (virtual-id space) is not wired yet.

What it means

When speculative decoding is enabled together with --enable-unified-memory, the resulting allocator is checked: if it is a UnifiedSWATokenToKVPoolAllocator the run is rejected, because the unified hybrid-SWA pool's draft sizing (virtual-id space) is not wired up yet. Only the hybrid-Mamba unified allocator supports a draft worker.

Source

Thrown at python/sglang/srt/mem_cache/kv_cache_configurator.py:441

            return _InitializedPools(
                req_to_token_pool=bundle.req_to_token_pool,
                token_to_kv_pool=bundle.token_to_kv_pool,
                token_to_kv_pool_allocator=bundle.token_to_kv_pool_allocator,
                unified_memory_pool=bundle.unified_memory_pool,
            )

        # The unified allocator hands out VIRTUAL token ids from the whole
        # virtual space (> max_total_num_tokens); the direct-indexed draft
        # pool must be sized by that space.
        draft_virtual_id_space: Optional[int] = None
        if self.is_draft_worker and token_to_kv_pool_allocator is not None:
            from sglang.srt.mem_cache.multi_ended_allocator import (
                UnifiedMambaTokenToKVPoolAllocator,
                UnifiedSWATokenToKVPoolAllocator,
            )

            if isinstance(token_to_kv_pool_allocator, UnifiedSWATokenToKVPoolAllocator):
                raise ValueError(
                    "Speculative decoding with --enable-unified-memory is only "
                    "supported for hybrid-Mamba targets; the unified hybrid-SWA "
                    "pool's draft sizing (virtual-id space) is not wired yet."
                )
            if isinstance(
                token_to_kv_pool_allocator, UnifiedMambaTokenToKVPoolAllocator
            ):
                draft_virtual_id_space = token_to_kv_pool_allocator.size_full
                assert draft_virtual_id_space >= sizes.max_total_num_tokens, (
                    "unified allocator virtual space smaller than the token "
                    f"budget: size_full={draft_virtual_id_space} < "
                    f"max_total_num_tokens={sizes.max_total_num_tokens}"
                )
                # Round UP to page alignment (paged draft backends view the
                # pool as (-1, page_size, H, D); size_full is not aligned).
                page = max(int(self.pool_page_size or 1), 1)
                draft_virtual_id_space = (
                    (draft_virtual_id_space + page - 1) // page * page

View on GitHub (pinned to 0132848349)

Solutions

  1. Disable speculative decoding for this model
  2. Drop --enable-unified-memory
  3. Use a hybrid-Mamba model where the unified allocator supports spec decode

Example fix

# before
python -m sglang.launch_server --model hybrid-swa --enable-unified-memory --speculative-algorithm EAGLE ...
# after
python -m sglang.launch_server --model hybrid-swa --enable-unified-memory ...
Defensive patterns

Strategy: validation

Validate before calling

if server_args.enable_unified_memory and server_args.speculative_algorithm and is_hybrid_swa(model_config):
    raise SystemExit("spec decode + unified-memory unsupported on hybrid-SWA; drop one")

Prevention

When it happens

Trigger: Launch with --enable-unified-memory plus a speculative decoding algorithm on a hybrid-SWA model; the allocator isinstance check in _init_pools fails.

Common situations: Enabling spec decode (e.g. EAGLE-style) on a hybrid-SWA model that uses the unified memory pool.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cb2a458448d957cf. Report an issue: GitHub.