sgl-project/sglang · error · ValueError

Not enough host memory for V4 paged pool {pool_name}. Reques

Error message

Not enough host memory for V4 paged pool {pool_name}. Requesting {requested_bytes / 1e9:.2f} GB but only have {available_bytes / 1e9:.2f} GB free.

What it means

The V4 paged host pool pre-checks that the requested host KV bytes (layer_num * num_host_pages * item_bytes) fit within the free host memory reported by host_memory_budget_bytes(); if not, it raises ValueError with the requested vs available GB.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool_host.py:212

        self.dtype = torch.uint8
        self.device = device
        self.pin_memory = pin_memory
        self.allocator = get_allocator_from_storage(allocator_type)
        self.page_size = slot_page_size
        self.size = num_host_pages * slot_page_size
        self.layout = layout
        self.size_per_token = item_bytes
        self.start_layer = 0
        self.end_layer = self.layer_num
        self.lock = threading.RLock()

        self.device_buffers = device_buffers
        self.gpu_device = device_buffers[0].device if device_buffers else device

        requested_bytes = self.layer_num * num_host_pages * self.item_bytes
        available_bytes = host_memory_budget_bytes()
        if requested_bytes > available_bytes:
            raise ValueError(
                f"Not enough host memory for V4 paged pool {pool_name}. "
                f"Requesting {requested_bytes / 1e9:.2f} GB but only have "
                f"{available_bytes / 1e9:.2f} GB free."
            )

        alloc_func = ALLOC_MEMORY_FUNCS[self.gpu_device]
        self.data_refs = []
        if self.layout == "layer_first":
            self.kv_buffer = [
                alloc_func(
                    (num_host_pages, self.item_bytes),
                    dtype=self.dtype,
                    device=self.device,
                    pin_memory=self.pin_memory,
                    allocator=self.allocator,
                )
                for _ in range(self.layer_num)
            ]

View on GitHub (pinned to 0132848349)

Solutions

Reduce the host pool size (lower hi-cache ratio / host page count config)
Free host memory or lower host_memory_budget_bytes cap; stop competing processes using RAM
Use fewer layers per pool or a smaller dtype/compression setting to shrink item_bytes
Verify actual free RAM (free -g) against the GB figures in the message and budget accordingly

Example fix

# before
host_ratio = 0.9  # pool requests more GB than free

# after
host_ratio = 0.4  # request fits within host_memory_budget_bytes()

Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.mem_cache.memory_pool_host import host_memory_budget_bytes
requested = layer_num * num_host_pages * item_bytes
assert requested <= host_memory_budget_bytes(), "shrink host pool config"

Type guard

null

Try / catch

try:
    pool = make_v4_pool(...)
except ValueError as e:
    if "Not enough host memory" in str(e):
        pool = make_v4_pool(..., num_host_pages=int(num_host_pages * 0.5))
    else:
        raise

Prevention

Check free RAM (free -g) before launching with large host-cache ratios
Start with a modest hi-cache ratio and scale up
Reserve headroom for other processes on shared hosts

When it happens

Trigger: Constructing a V4 paged host pool whose computed size exceeds current free CPU RAM — e.g. large --max-mamba-cache-size / hi-cache ratio, big layer counts, or a machine where other processes already consumed most host memory.

Common situations: Aggressive hierarchical-cache (HiCache) host memory ratios on small-RAM machines; changing host memory budget env vars; running multiple servers on one host so each sees less free memory; larger models (more layers) with unchanged page counts.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ee0654df696cc77b. Report an issue: GitHub.