sgl-project/sglang · error · ValueError

--hicache-host-memory-mode buffer_only on SWA models require

Error message

--hicache-host-memory-mode buffer_only on SWA models requires an SWA host staging pool; the unified_kv layout keeps SWA as a device-only ring.

What it means

Thrown when buffer_only host memory mode is used with a sliding-window-attention (SWA) model that has no SWA host staging pool (unified_kv layout keeps SWA as a device-only ring). Without a host pool the SWA window can neither stage writes nor fetch load-backs, so the stack refuses to start.

Source

Thrown at python/sglang/srt/mem_cache/buffer_mode/pipeline.py:168

def validate_buffer_only_stack(
    sidecar_pool_specs: list, swa_component: Optional[SWAComponent]
) -> None:
    """Post-assembly buffer-mode fences.

    Sidecar pools (DSv4 compressed regions) and unified_kv SWA (device-only
    ring, never offloaded) have no per-pool staging path yet.
    """
    if sidecar_pool_specs:
        raise ValueError(
            "--hicache-host-memory-mode buffer_only does not support "
            "sidecar storage pools (DeepSeek-V4 compressed regions)."
        )
    swa = swa_component
    if swa is not None and swa._swa_kv_pool_host is None:
        # Only reachable on SWA models with the unified_kv layout (SWA as
        # a device-only ring): without a host pool the window can neither
        # stage for writes nor fetch for load-backs.
        raise ValueError(
            "--hicache-host-memory-mode buffer_only on SWA models "
            "requires an SWA host staging pool; the unified_kv layout "
            "keeps SWA as a device-only ring."
        )
    if swa is not None and swa._swa_kv_pool_host is not None:
        # Below two windows the pool cannot hold a staging write AND the
        # loads-priority reserve (_aux_loads_margin floors at one
        # window), so every window-carrying intent would be dropped as
        # oversize and SWA storage coverage would silently be zero.
        window_tokens = swa.full_window_pages * swa._swa_kv_pool_host.page_size
        if swa._swa_kv_pool_host.size < 2 * window_tokens:
            raise ValueError(
                "--hicache-host-memory-mode buffer_only requires an SWA "
                f"host pool of at least two trailing windows "
                f"({2 * window_tokens} tokens; got "
                f"{swa._swa_kv_pool_host.size}): one staging a write "
                "while one stays reserved for prefetch window allocs."
            )

View on GitHub (pinned to 0132848349)

Solutions

  1. Disable buffer_only mode for SWA models (use default hicache host memory mode that creates the SWA host pool)
  2. Configure the run so the SWA component gets a host staging pool (enable SWA offload / non-unified layout)
  3. Use a non-SWA model if buffer_only is mandatory

Example fix

# before
--hicache-host-memory-mode buffer_only   # SWA model, unified_kv layout
# after
# (drop the flag, or enable SWA host offload so _swa_kv_pool_host exists)
Defensive patterns

Strategy: validation

Validate before calling

def swa_host_pool_ok(swa_comp):
    return swa_comp is None or swa_comp._swa_kv_pool_host is not None
# in launch config: if mode == 'buffer_only' and not swa_host_pool_ok(swa): use default mode

Type guard

def has_swa_host_pool(swa_comp) -> bool:
    return swa_comp is not None and getattr(swa_comp, "_swa_kv_pool_host", None) is not None

Prevention

When it happens

Trigger: Server launch with --hicache-host-memory-mode buffer_only on an SWA model using the unified_kv layout where swa_component is non-None but _swa_kv_pool_host is None; validate_buffer_only_stack raises inside init_hicache.

Common situations: Running buffer_only mode on SWA/hybrid models (e.g. Gemma-style or other sliding window architectures) with the unified_kv pool layout; changing kv layout settings that disable SWA host offload.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/680c3b20abf0e9f7. Report an issue: GitHub.