{"record":{"id":"d1f8484c734becbb","repo":"sgl-project/sglang","slug":"hicache-host-memory-mode-buffer-only-requires-an","errorCode":null,"errorMessage":"--hicache-host-memory-mode buffer_only requires an SWA host pool of at least two trailing windows ({2 * window_tokens} tokens; got {swa._swa_kv_pool_host.size}): one staging a write while one stays reserved for prefetch window allocs.","messagePattern":"--hicache-host-memory-mode buffer_only requires an SWA host pool of at least two trailing windows \\((.+?) tokens; got (.+?)\\): one staging a write while one stays reserved for prefetch window allocs\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/buffer_mode/pipeline.py","lineNumber":180,"sourceCode":"        )\n    swa = swa_component\n    if swa is not None and swa._swa_kv_pool_host is None:\n        # Only reachable on SWA models with the unified_kv layout (SWA as\n        # a device-only ring): without a host pool the window can neither\n        # stage for writes nor fetch for load-backs.\n        raise ValueError(\n            \"--hicache-host-memory-mode buffer_only on SWA models \"\n            \"requires an SWA host staging pool; the unified_kv layout \"\n            \"keeps SWA as a device-only ring.\"\n        )\n    if swa is not None and swa._swa_kv_pool_host is not None:\n        # Below two windows the pool cannot hold a staging write AND the\n        # loads-priority reserve (_aux_loads_margin floors at one\n        # window), so every window-carrying intent would be dropped as\n        # oversize and SWA storage coverage would silently be zero.\n        window_tokens = swa.full_window_pages * swa._swa_kv_pool_host.page_size\n        if swa._swa_kv_pool_host.size < 2 * window_tokens:\n            raise ValueError(\n                \"--hicache-host-memory-mode buffer_only requires an SWA \"\n                f\"host pool of at least two trailing windows \"\n                f\"({2 * window_tokens} tokens; got \"\n                f\"{swa._swa_kv_pool_host.size}): one staging a write \"\n                \"while one stays reserved for prefetch window allocs.\"\n            )\n\n\nclass BufferModePipeline:\n    \"\"\"All buffer-mode state plus the backup and load-back pipelines.\n\n    Constructed by ``UnifiedRadixCache.init_hicache`` when host memory mode\n    is ``buffer_only``; ``cache.buffer_pipeline is None`` elsewhere, which\n    the cache's mode branches use as the dispatch test.\n    \"\"\"\n\n    def __init__(\n        self,","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/buffer_mode/pipeline.py#L162-L198","documentation":"Buffer-only mode requires the SWA host pool to hold at least two trailing windows of tokens: one to stage a write while one stays reserved for prefetch window allocations. If the pool is smaller, every window-carrying intent would be dropped as oversize and SWA storage coverage would silently be zero, so the size is validated at startup.","triggerScenarios":"SWA model with a host pool but swa._swa_kv_pool_host.size < 2 * window_tokens where window_tokens = swa.full_window_pages * page_size; typically a small --hicache-swa-host-memory-size or similar sizing flag.","commonSituations":"Setting an aggressive small SWA host pool size to save RAM; using a long sliding window (large full_window_pages) with a host pool sized for the default window; version changes that alter window accounting.","solutions":["Increase the SWA host pool size to at least 2 * window_tokens (e.g. raise --hicache-swa-host-memory-size / --swa-host-memory-size flag)","Reduce the sliding window size if the model/config allows it","Disable buffer_only mode so the two-window reserve is not required"],"exampleFix":"# before: pool sized for ~1 window\n--hicache-host-memory-mode buffer_only --hicache-swa-host-memory-size 100000000\n# after: pool sized >= 2 * full_window_pages * page_size\n--hicache-host-memory-mode buffer_only --hicache-swa-host-memory-size 500000000","handlingStrategy":"validation","validationCode":"window_tokens = swa.full_window_pages * swa._swa_kv_pool_host.page_size\nassert swa._swa_kv_pool_host.size >= 2 * window_tokens, \"SWA host pool too small for buffer_only\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size SWA host pools as a multiple of the sliding window, never a round byte number","Add a pre-launch check computing required = 2 * window_pages * page_size","Watch for this when changing sliding-window length in model configs"],"tags":["hicache","buffer-only","swa","memory-sizing","validation"],"backgroundTag":"memory-pool-undersized","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}