sgl-project/sglang · error · ValueError

--hicache-host-memory-mode buffer_only requires an SWA host…

Error message

--hicache-host-memory-mode buffer_only requires an SWA host pool of at least two trailing windows ({2 * window_tokens} tokens; got {swa._swa_kv_pool_host.size}): one staging a write while one stays reserved for prefetch window allocs.

What it means

Buffer-only mode requires the SWA host pool to hold at least two trailing windows of tokens: one to stage a write while one stays reserved for prefetch window allocations. If the pool is smaller, every window-carrying intent would be dropped as oversize and SWA storage coverage would silently be zero, so the size is validated at startup.

Solutions

  1. Increase the SWA host pool size to at least 2 * window_tokens (e.g. raise --hicache-swa-host-memory-size / --swa-host-memory-size flag)
  2. Reduce the sliding window size if the model/config allows it
  3. Disable buffer_only mode so the two-window reserve is not required

Example fix

# before: pool sized for ~1 window
--hicache-host-memory-mode buffer_only --hicache-swa-host-memory-size 100000000
# after: pool sized >= 2 * full_window_pages * page_size
--hicache-host-memory-mode buffer_only --hicache-swa-host-memory-size 500000000
Defensive patterns

Strategy: validation

Validate before calling

window_tokens = swa.full_window_pages * swa._swa_kv_pool_host.page_size
assert swa._swa_kv_pool_host.size >= 2 * window_tokens, "SWA host pool too small for buffer_only"

Prevention

When it happens

Trigger: SWA model with a host pool but swa._swa_kv_pool_host.size < 2 * window_tokens where window_tokens = swa.full_window_pages * page_size; typically a small --hicache-swa-host-memory-size or similar sizing flag.

Common situations: Setting an aggressive small SWA host pool size to save RAM; using a long sliding window (large full_window_pages) with a host pool sized for the default window; version changes that alter window accounting.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d1f8484c734becbb. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/mem_cache/buffer_mode/pipeline.py:180

        )
    swa = swa_component
    if swa is not None and swa._swa_kv_pool_host is None:
        # Only reachable on SWA models with the unified_kv layout (SWA as
        # a device-only ring): without a host pool the window can neither
        # stage for writes nor fetch for load-backs.
        raise ValueError(
            "--hicache-host-memory-mode buffer_only on SWA models "
            "requires an SWA host staging pool; the unified_kv layout "
            "keeps SWA as a device-only ring."
        )
    if swa is not None and swa._swa_kv_pool_host is not None:
        # Below two windows the pool cannot hold a staging write AND the
        # loads-priority reserve (_aux_loads_margin floors at one
        # window), so every window-carrying intent would be dropped as
        # oversize and SWA storage coverage would silently be zero.
        window_tokens = swa.full_window_pages * swa._swa_kv_pool_host.page_size
        if swa._swa_kv_pool_host.size < 2 * window_tokens:
            raise ValueError(
                "--hicache-host-memory-mode buffer_only requires an SWA "
                f"host pool of at least two trailing windows "
                f"({2 * window_tokens} tokens; got "
                f"{swa._swa_kv_pool_host.size}): one staging a write "
                "while one stays reserved for prefetch window allocs."
            )


class BufferModePipeline:
    """All buffer-mode state plus the backup and load-back pipelines.

    Constructed by ``UnifiedRadixCache.init_hicache`` when host memory mode
    is ``buffer_only``; ``cache.buffer_pipeline is None`` elsewhere, which
    the cache's mode branches use as the dispatch test.
    """

    def __init__(
        self,

View on GitHub (pinned to 0132848349)