{"record":{"id":"452006b5ad686a05","repo":"sgl-project/sglang","slug":"decode-out-of-memory-try-to-lower-your-batch-size","errorCode":null,"errorMessage":"Decode out of memory. Try to lower your batch size.\\nTry to allocate {len(seq_lens) * token_per_req} tokens.\\n{available_and_evictable_str(tree_cache)}","messagePattern":"Decode out of memory\\. Try to lower your batch size\\.\\\\nTry to allocate (.+?) tokens\\.\\\\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/allocation.py","lineNumber":523,"sourceCode":"\n    if is_dsv4:\n        bundle = out\n        out_cache_loc = None if bundle is None else bundle.out_full_loc\n        if batch is not None:\n            batch.out_cache_loc_dsv4 = bundle\n    else:\n        out_cache_loc = out\n\n    if out_cache_loc is None:\n        error_msg = (\n            f\"Decode out of memory. Try to lower your batch size.\\n\"\n            f\"Try to allocate {len(seq_lens) * token_per_req} tokens.\\n\"\n            f\"{available_and_evictable_str(tree_cache)}\"\n        )\n        logger.error(error_msg)\n        if tree_cache is not None:\n            tree_cache.pretty_print()\n        raise RuntimeError(error_msg)\n\n    return out_cache_loc\n\n\ndef alloc_for_decode(batch: ScheduleBatch, token_per_req: int) -> torch.Tensor:\n    \"\"\"\n    Allocate KV cache for decode batch and write to req_to_token_pool.\n\n    Returns:\n        out_cache_loc: allocated cache locations\n    \"\"\"\n\n    batch.maybe_evict_swa()\n\n    seq_lens_gpu = batch.seq_lens\n    bs = seq_lens_gpu.shape[0]\n\n    if _alloc_page_size(batch) == 1:","sourceCodeStart":505,"sourceCodeEnd":541,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/allocation.py#L505-L541","documentation":"The paged-KV decode allocation (alloc_paged_token_slots_decode) cannot allocate len(seq_lens) * token_per_req slots (one new page-slot per sequence per decode step, times tokens per request for spec-decode/multi-token decode). All running sequences must grow their KV every step, so a full pool aborts the batch.","triggerScenarios":"Decode-step allocation on a paged KV cache when running batch size × token_per_req exceeds free + evictable pages; aggravated by speculative decoding where token_per_req > 1, or by very large running batches.","commonSituations":"Large --max-running-requests during long decode phases filling pages; spec-decode draft tokens multiplying per-step allocation; small mem-fraction leaving too few pages; SWA/hybrid models fragmenting pages.","solutions":["Lower --max-running-requests (directly shrinks len(seq_lens))","Increase --mem-fraction-static","If using speculative decoding, reduce num speculative tokens or disable it","Reduce --context-length so fewer pages are held per sequence"],"exampleFix":"# before\n--max-running-requests 512 --speculative-num-steps 5\n# after\n--max-running-requests 128 --speculative-num-steps 3","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    token = await client.generate(prompt, sampling_params, stream=True)\nexcept RuntimeError as e:\n    if \"Decode out of memory\" in str(e):\n        await asyncio.sleep(backoff); retry()  # scheduler retries after retraction\n    raise","preventionTips":["Leave headroom in --max-running-requests for decode-phase page growth","Tune speculative decoding token counts against page availability","Watch KV usage over full generation length, not just prefill"],"tags":["sglang","memory","decode","paged-kv"],"backgroundTag":"kv-cache-out-of-memory","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}