sgl-project/sglang · critical · RuntimeError

alloc_req_slots runs out of memory. Please set a smaller num

Error message

alloc_req_slots runs out of memory. Please set a smaller number for `--max-running-requests`. {req_to_token_pool.available_size()=}, {num_reqs=}, 

What it means

The req_to_token_pool (per-request metadata slots mapping requests to token indices) has fewer available slots than the num_reqs being scheduled. This pool is sized by max-running-requests times context length, so it caps how many concurrent requests the scheduler can track.

Source

Thrown at python/sglang/srt/mem_cache/allocation.py:265

        # Eviction headroom factor: 3x (or lazy variant) for radix COW, 1x for chunk.
        if tree_cache.supports_mamba():
            factor = (
                MAMBA_STATE_PER_REQ_PREFIX_CACHE_LAZY
                if req_to_token_pool.enable_mamba_extra_buffer_lazy
                else MAMBA_STATE_PER_REQ_PREFIX_CACHE
            )
        else:
            factor = MAMBA_STATE_PER_REQ_NO_CACHE
        mamba_state_needed = num_reqs * factor
        if mamba_available_size < mamba_state_needed:
            if tree_cache is not None and tree_cache.supports_mamba():
                mamba_num = max(0, mamba_state_needed - mamba_available_size)
                tree_cache.evict_for_alloc(
                    EvictParams(num_tokens=0, mamba_num=mamba_num)
                )
    req_pool_indices = req_to_token_pool.alloc(reqs)
    if req_pool_indices is None:
        raise RuntimeError(
            "alloc_req_slots runs out of memory. "
            "Please set a smaller number for `--max-running-requests`. "
            f"{req_to_token_pool.available_size()=}, {num_reqs=}, "
        )
    return req_pool_indices


def _alloc_page_size(batch: ScheduleBatch) -> int:
    # DCP swaps in an allocator whose page_size is the configured page_size *
    # dcp_size, so it can be > 1 even when tree_cache.page_size is 1; branch on
    # the real allocator's page_size there. Elsewhere the two are equal.
    if (_is_hip or _is_cuda) and get_parallel().dcp_enabled:
        return batch.tree_cache.token_to_kv_pool_allocator.page_size
    return batch.tree_cache.page_size


def alloc_for_extend(
    batch: ScheduleBatch,

View on GitHub (pinned to 0132848349)

Solutions

1. Lower --max-running-requests to at or below req_to_token_pool.available_size()
2. Increase --context-length so the req-to-token pool is allocated larger (pool size derives from context length)
3. Reduce concurrent request pressure in the client

Example fix

# before
--max-running-requests 1024 --context-length 4096
# after
--max-running-requests 256 --context-length 8192

avail = server.req_to_token_pool.available_size()  # introspection variant
submitted = min(len(batch), max_running_requests)

try:
    ...
except RuntimeError as e:
    if "alloc_req_slots" in str(e):
        lower_max_running_requests_and_restart()

When it happens

Trigger: alloc_for_extend tries req_to_token_pool.alloc(reqs) for num_reqs exceeding available_size(); typically when --max-running-requests is set higher than the pool was sized for, or the pool is smaller due to a small --context-length or --max-prefill-items style sizing.

Common situations: Raising --max-running-requests without resizing the req pool; hybrid/mamba models consuming extra req slots; long context splitting reducing effective pool size.

Related errors

• top_logprobs_num {top_logprobs_len} exceeds disaggregation m
• kv-canary: forward_batch.batch_size={bs} exceeds pre-allocat
• kv-canary: forward_batch token count={num_tokens} exceeds pr
• Failed to load LoRA adapter {lora_ref.lora_name} as a pinned
• Out of memory. Try to lower your batch size.\nTry to allocat

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5005b2b87598a62d. Report an issue: GitHub.