sgl-project/sglang · critical · ValueError

Not enough host memory for DSA indexer hierarchical cache. R

Error message

Not enough host memory for DSA indexer hierarchical cache. Requesting {requested_bytes / 1e9:.2f} GB but only have {available_bytes / 1e9:.2f} GB free.

What it means

The DSA (DeepSeek Sparse Attention) indexer hierarchical cache host pool sizes its index buffer from page_num * layer_num * page stride * dtype itemsize, and checks it against the free host memory budget before allocating. If the required buffer exceeds available host RAM it refuses to construct. This protects the process from an OOM kill at allocation time.

Source

Thrown at python/sglang/srt/mem_cache/pool_host/dsa.py:99

            + self.index_head_dim // self.indexer_quant_block_size * 4
        )
        self.size = anchor_host.size
        self.page_num = anchor_host.page_num

        self.indexer_page_stride_size = (
            self.indexer_size_per_token * self.page_size * self.indexer_dtype.itemsize
        )
        self.indexer_layout_dim = self.indexer_page_stride_size * self.layer_num
        self.indexer_page_num = (self.size + self.page_size + 1) // self.page_size
        self.size_per_token = (
            self.indexer_size_per_token * self.layer_num * self.indexer_dtype.itemsize
        )

        buf_elem_size = self.page_num * self.layer_num * self.indexer_page_stride_size
        requested_bytes = buf_elem_size * self.indexer_dtype.itemsize
        available_bytes = host_memory_budget_bytes()
        if requested_bytes > available_bytes:
            raise ValueError(
                f"Not enough host memory for DSA indexer hierarchical cache. "
                f"Requesting {requested_bytes / 1e9:.2f} GB but only have "
                f"{available_bytes / 1e9:.2f} GB free."
            )
        draft_layer_num = self.layer_num - self.target_layer_num
        if draft_layer_num > 0:
            logger.info(
                "Allocating %.2f GB host memory for DSA indexer (layout=%s), "
                "packed MTP layers: "
                "target_layers=%d, draft_layers=%d, total_layers=%d.",
                requested_bytes / 1e9,
                layout,
                self.target_layer_num,
                draft_layer_num,
                self.layer_num,
            )
        else:
            logger.info(

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce hierarchical cache size (e.g. lower --hp-size / max host pages)
2. Free host memory or increase the machine's RAM / cgroup limit
3. Use a smaller indexer dtype if supported
4. Compute page_num*layer_num*stride*itemsize up front and validate against psutil free memory before launching

Example fix

# before
server_args.hierarchical_cache_size_gb = 200  # exceeds free RAM
# after
server_args.hierarchical_cache_size_gb = 50   # fit within host_memory_budget_bytes()

import psutil
requested = page_num * layer_num * page_stride * dtype_itemsize
avail = psutil.virtual_memory().available
assert requested <= avail, f"need {requested/1e9:.1f}GB, have {avail/1e9:.1f}GB"

When it happens

Trigger: Constructing DSATokenToKVPoolHost (dsa.py __init__) with a large --hp-size / page count, many layers, or a wide indexer dtype on a machine whose host_memory_budget_bytes() reports less free memory than the buffer needs.

Common situations: Setting a very large hierarchical cache size (hp-size) relative to physical RAM; running alongside other processes consuming host memory; fp8 vs bf16 indexer dtype changes altering itemsize.

Related errors

• Component {component_name!r} resolved to layerwise-offload,
• Failed to get server info. {error_data['error']['message']}
• world_size ({world_size}) is less than tensor_parallel_degre
• Could not connect to remote scheduler at {self.server_args.s
• batching config {source} does not contain any rules

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/92dbea609232f1bc. Report an issue: GitHub.