{"record":{"id":"5005b2b87598a62d","repo":"sgl-project/sglang","slug":"alloc-req-slots-runs-out-of-memory-please-set-a-s","errorCode":null,"errorMessage":"alloc_req_slots runs out of memory. Please set a smaller number for `--max-running-requests`. {req_to_token_pool.available_size()=}, {num_reqs=}, ","messagePattern":"alloc_req_slots runs out of memory\\. Please set a smaller number for `--max-running-requests`\\. (.+?), (.+?), ","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/allocation.py","lineNumber":265,"sourceCode":"        # Eviction headroom factor: 3x (or lazy variant) for radix COW, 1x for chunk.\n        if tree_cache.supports_mamba():\n            factor = (\n                MAMBA_STATE_PER_REQ_PREFIX_CACHE_LAZY\n                if req_to_token_pool.enable_mamba_extra_buffer_lazy\n                else MAMBA_STATE_PER_REQ_PREFIX_CACHE\n            )\n        else:\n            factor = MAMBA_STATE_PER_REQ_NO_CACHE\n        mamba_state_needed = num_reqs * factor\n        if mamba_available_size < mamba_state_needed:\n            if tree_cache is not None and tree_cache.supports_mamba():\n                mamba_num = max(0, mamba_state_needed - mamba_available_size)\n                tree_cache.evict_for_alloc(\n                    EvictParams(num_tokens=0, mamba_num=mamba_num)\n                )\n    req_pool_indices = req_to_token_pool.alloc(reqs)\n    if req_pool_indices is None:\n        raise RuntimeError(\n            \"alloc_req_slots runs out of memory. \"\n            \"Please set a smaller number for `--max-running-requests`. \"\n            f\"{req_to_token_pool.available_size()=}, {num_reqs=}, \"\n        )\n    return req_pool_indices\n\n\ndef _alloc_page_size(batch: ScheduleBatch) -> int:\n    # DCP swaps in an allocator whose page_size is the configured page_size *\n    # dcp_size, so it can be > 1 even when tree_cache.page_size is 1; branch on\n    # the real allocator's page_size there. Elsewhere the two are equal.\n    if (_is_hip or _is_cuda) and get_parallel().dcp_enabled:\n        return batch.tree_cache.token_to_kv_pool_allocator.page_size\n    return batch.tree_cache.page_size\n\n\ndef alloc_for_extend(\n    batch: ScheduleBatch,","sourceCodeStart":247,"sourceCodeEnd":283,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/allocation.py#L247-L283","documentation":"The req_to_token_pool (per-request metadata slots mapping requests to token indices) has fewer available slots than the num_reqs being scheduled. This pool is sized by max-running-requests times context length, so it caps how many concurrent requests the scheduler can track.","triggerScenarios":"alloc_for_extend tries req_to_token_pool.alloc(reqs) for num_reqs exceeding available_size(); typically when --max-running-requests is set higher than the pool was sized for, or the pool is smaller due to a small --context-length or --max-prefill-items style sizing.","commonSituations":"Raising --max-running-requests without resizing the req pool; hybrid/mamba models consuming extra req slots; long context splitting reducing effective pool size.","solutions":["Lower --max-running-requests to at or below req_to_token_pool.available_size()","Increase --context-length so the req-to-token pool is allocated larger (pool size derives from context length)","Reduce concurrent request pressure in the client"],"exampleFix":"# before\n--max-running-requests 1024 --context-length 4096\n# after\n--max-running-requests 256 --context-length 8192","handlingStrategy":"validation","validationCode":"avail = server.req_to_token_pool.available_size()  # introspection variant\nsubmitted = min(len(batch), max_running_requests)","typeGuard":null,"tryCatchPattern":"try:\n    ...\nexcept RuntimeError as e:\n    if \"alloc_req_slots\" in str(e):\n        lower_max_running_requests_and_restart()","preventionTips":["Keep client concurrency at or below --max-running-requests","Pair increases in max-running-requests with matching req pool sizing (context length)","Verify pool sizing at startup from logs"],"tags":["sglang","memory","request-pool","capacity"],"backgroundTag":"request-slot-exhaustion","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}