{"record":{"id":"4c7e8eb4d955c1d3","repo":"sgl-project/sglang","slug":"out-of-memory-try-to-lower-your-batch-size-ntry","errorCode":null,"errorMessage":"Out of memory. Try to lower your batch size.\\nTry to allocate {num_tokens} tokens.\\n{available_and_evictable_str(tree_cache)}","messagePattern":"Out of memory\\. Try to lower your batch size\\.\\\\nTry to allocate (.+?) tokens\\.\\\\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/allocation.py","lineNumber":168,"sourceCode":"def alloc_token_slots(\n    tree_cache: BasePrefixCache,\n    num_tokens: int,\n):\n    allocator = tree_cache.token_to_kv_pool_allocator\n    evict_from_tree_cache(tree_cache, num_tokens)\n\n    out_cache_loc = allocator.alloc(num_tokens)\n\n    if out_cache_loc is None:\n        error_msg = (\n            f\"Out of memory. Try to lower your batch size.\\n\"\n            f\"Try to allocate {num_tokens} tokens.\\n\"\n            f\"{available_and_evictable_str(tree_cache)}\"\n        )\n        logger.error(error_msg)\n        if tree_cache is not None:\n            tree_cache.pretty_print()\n        raise RuntimeError(error_msg)\n\n    return out_cache_loc\n\n\ndef alloc_paged_token_slots_extend(\n    tree_cache: BasePrefixCache,\n    prefix_lens: torch.Tensor,\n    prefix_lens_cpu: torch.Tensor,\n    seq_lens: torch.Tensor,\n    seq_lens_cpu: torch.Tensor,\n    last_loc: torch.Tensor,\n    extend_num_tokens: int,\n    req_pool_indices: Optional[torch.Tensor] = None,\n    batch=None,\n):\n    # Over estimate the number of tokens: assume each request needs a new page.\n    allocator = tree_cache.token_to_kv_pool_allocator\n    num_tokens = extend_num_tokens + len(seq_lens_cpu) * allocator.page_size","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/allocation.py#L150-L186","documentation":"alloc_token_slots could not find num_tokens free KV-cache slots even after considering evictable prefix-cache entries, so the request cannot be admitted. The message reports how many tokens were needed and how much memory (including evictable cached prefixes) was available.","triggerScenarios":"A batch's token allocation (extend/decode/spec-decode, or KV-reuse paths) exceeds token_to_kv_pool available + evictable size — typically very long prompts, huge batch sizes, or a small --mem-fraction-static leaving a tiny KV pool.","commonSituations":"Long-context requests with aggressive prefix caching filling the pool; max-running-requests too high; context length close to KV capacity; memory squeezed by a large model or HiCache/reserved buffers.","solutions":["Reduce --max-running-requests or batch size","Lower individual request length / --context-length","Increase --mem-fraction-static so the KV pool is larger","Clear or shrink the prefix cache (--chunked-prefill-size tuning, or disable prefix caching to avoid evictable-vs-usable confusion)","Restart with a smaller model or tensor parallelism to free per-GPU KV memory"],"exampleFix":"# before\npython -m sglang.launch_server --model meta-llama/Llama-3-70B --max-running-requests 256\n# after\npython -m sglang.launch_server --model meta-llama/Llama-3-70B --max-running-requests 64 --mem-fraction-static 0.9","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    out = engine.generate(prompt, sampling_params)\nexcept RuntimeError as e:\n    if \"Out of memory\" in str(e):\n        reduce_batch_and_retry()\n    else:\n        raise","preventionTips":["Size --mem-fraction-static, --max-running-requests, and --context-length to worst-case token demand","Load-test with realistic prompt lengths before serving","Monitor KV pool utilization metrics and shed load proactively"],"tags":["sglang","memory","kv-cache","allocation"],"backgroundTag":"kv-cache-out-of-memory","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}