sgl-project/sglang · error · ValueError

--hicache-host-memory-mode buffer_only requires an SWA host

Error message

--hicache-host-memory-mode buffer_only requires an SWA host pool of at least two trailing windows ({2 * window_tokens} tokens; got {swa._swa_kv_pool_host.size}): one staging a write while one stays reserved for prefetch window allocs.

What it means

Buffer-only mode requires the SWA host pool to hold at least two trailing windows of tokens: one to stage a write while one stays reserved for prefetch window allocations. If the pool is smaller, every window-carrying intent would be dropped as oversize and SWA storage coverage would silently be zero, so the size is validated at startup.

Source

Thrown at python/sglang/srt/mem_cache/buffer_mode/pipeline.py:180

        )
    swa = swa_component
    if swa is not None and swa._swa_kv_pool_host is None:
        # Only reachable on SWA models with the unified_kv layout (SWA as
        # a device-only ring): without a host pool the window can neither
        # stage for writes nor fetch for load-backs.
        raise ValueError(
            "--hicache-host-memory-mode buffer_only on SWA models "
            "requires an SWA host staging pool; the unified_kv layout "
            "keeps SWA as a device-only ring."
        )
    if swa is not None and swa._swa_kv_pool_host is not None:
        # Below two windows the pool cannot hold a staging write AND the
        # loads-priority reserve (_aux_loads_margin floors at one
        # window), so every window-carrying intent would be dropped as
        # oversize and SWA storage coverage would silently be zero.
        window_tokens = swa.full_window_pages * swa._swa_kv_pool_host.page_size
        if swa._swa_kv_pool_host.size < 2 * window_tokens:
            raise ValueError(
                "--hicache-host-memory-mode buffer_only requires an SWA "
                f"host pool of at least two trailing windows "
                f"({2 * window_tokens} tokens; got "
                f"{swa._swa_kv_pool_host.size}): one staging a write "
                "while one stays reserved for prefetch window allocs."
            )


class BufferModePipeline:
    """All buffer-mode state plus the backup and load-back pipelines.

    Constructed by ``UnifiedRadixCache.init_hicache`` when host memory mode
    is ``buffer_only``; ``cache.buffer_pipeline is None`` elsewhere, which
    the cache's mode branches use as the dispatch test.
    """

    def __init__(
        self,

View on GitHub (pinned to 0132848349)

Solutions

Increase the SWA host pool size to at least 2 * window_tokens (e.g. raise --hicache-swa-host-memory-size / --swa-host-memory-size flag)
Reduce the sliding window size if the model/config allows it
Disable buffer_only mode so the two-window reserve is not required

Example fix

# before: pool sized for ~1 window
--hicache-host-memory-mode buffer_only --hicache-swa-host-memory-size 100000000
# after: pool sized >= 2 * full_window_pages * page_size
--hicache-host-memory-mode buffer_only --hicache-swa-host-memory-size 500000000

Defensive patterns

Strategy: validation

Validate before calling

window_tokens = swa.full_window_pages * swa._swa_kv_pool_host.page_size
assert swa._swa_kv_pool_host.size >= 2 * window_tokens, "SWA host pool too small for buffer_only"

Prevention

Size SWA host pools as a multiple of the sliding window, never a round byte number
Add a pre-launch check computing required = 2 * window_pages * page_size
Watch for this when changing sliding-window length in model configs

When it happens

Trigger: SWA model with a host pool but swa._swa_kv_pool_host.size < 2 * window_tokens where window_tokens = swa.full_window_pages * page_size; typically a small --hicache-swa-host-memory-size or similar sizing flag.

Common situations: Setting an aggressive small SWA host pool size to save RAM; using a long sliding window (large full_window_pages) with a host pool sized for the default window; version changes that alter window accounting.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d1f8484c734becbb. Report an issue: GitHub.