{"record":{"id":"ee0654df696cc77b","repo":"sgl-project/sglang","slug":"not-enough-host-memory-for-v4-paged-pool-pool-nam","errorCode":null,"errorMessage":"Not enough host memory for V4 paged pool {pool_name}. Requesting {requested_bytes / 1e9:.2f} GB but only have {available_bytes / 1e9:.2f} GB free.","messagePattern":"Not enough host memory for V4 paged pool (.+?)\\. Requesting (.+?) GB but only have (.+?) GB free\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/memory_pool_host.py","lineNumber":212,"sourceCode":"        self.dtype = torch.uint8\n        self.device = device\n        self.pin_memory = pin_memory\n        self.allocator = get_allocator_from_storage(allocator_type)\n        self.page_size = slot_page_size\n        self.size = num_host_pages * slot_page_size\n        self.layout = layout\n        self.size_per_token = item_bytes\n        self.start_layer = 0\n        self.end_layer = self.layer_num\n        self.lock = threading.RLock()\n\n        self.device_buffers = device_buffers\n        self.gpu_device = device_buffers[0].device if device_buffers else device\n\n        requested_bytes = self.layer_num * num_host_pages * self.item_bytes\n        available_bytes = host_memory_budget_bytes()\n        if requested_bytes > available_bytes:\n            raise ValueError(\n                f\"Not enough host memory for V4 paged pool {pool_name}. \"\n                f\"Requesting {requested_bytes / 1e9:.2f} GB but only have \"\n                f\"{available_bytes / 1e9:.2f} GB free.\"\n            )\n\n        alloc_func = ALLOC_MEMORY_FUNCS[self.gpu_device]\n        self.data_refs = []\n        if self.layout == \"layer_first\":\n            self.kv_buffer = [\n                alloc_func(\n                    (num_host_pages, self.item_bytes),\n                    dtype=self.dtype,\n                    device=self.device,\n                    pin_memory=self.pin_memory,\n                    allocator=self.allocator,\n                )\n                for _ in range(self.layer_num)\n            ]","sourceCodeStart":194,"sourceCodeEnd":230,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/memory_pool_host.py#L194-L230","documentation":"The V4 paged host pool pre-checks that the requested host KV bytes (layer_num * num_host_pages * item_bytes) fit within the free host memory reported by host_memory_budget_bytes(); if not, it raises ValueError with the requested vs available GB.","triggerScenarios":"Constructing a V4 paged host pool whose computed size exceeds current free CPU RAM — e.g. large --max-mamba-cache-size / hi-cache ratio, big layer counts, or a machine where other processes already consumed most host memory.","commonSituations":"Aggressive hierarchical-cache (HiCache) host memory ratios on small-RAM machines; changing host memory budget env vars; running multiple servers on one host so each sees less free memory; larger models (more layers) with unchanged page counts.","solutions":["Reduce the host pool size (lower hi-cache ratio / host page count config)","Free host memory or lower host_memory_budget_bytes cap; stop competing processes using RAM","Use fewer layers per pool or a smaller dtype/compression setting to shrink item_bytes","Verify actual free RAM (free -g) against the GB figures in the message and budget accordingly"],"exampleFix":"# before\nhost_ratio = 0.9  # pool requests more GB than free\n\n# after\nhost_ratio = 0.4  # request fits within host_memory_budget_bytes()","handlingStrategy":"validation","validationCode":"from sglang.srt.mem_cache.memory_pool_host import host_memory_budget_bytes\nrequested = layer_num * num_host_pages * item_bytes\nassert requested <= host_memory_budget_bytes(), \"shrink host pool config\"","typeGuard":"null","tryCatchPattern":"try:\n    pool = make_v4_pool(...)\nexcept ValueError as e:\n    if \"Not enough host memory\" in str(e):\n        pool = make_v4_pool(..., num_host_pages=int(num_host_pages * 0.5))\n    else:\n        raise","preventionTips":["Check free RAM (free -g) before launching with large host-cache ratios","Start with a modest hi-cache ratio and scale up","Reserve headroom for other processes on shared hosts"],"tags":["sglang","host-memory","hicache","out-of-memory","capacity"],"backgroundTag":"host-memory-budget-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}