{"record":{"id":"0aa273818a8c7e29","repo":"sgl-project/sglang","slug":"prefill-out-of-memory-try-to-lower-your-batch-siz","errorCode":null,"errorMessage":"Prefill out of memory. Try to lower your batch size.\\nTry to allocate {extend_num_tokens} tokens.\\n{available_and_evictable_str(tree_cache)}","messagePattern":"Prefill out of memory\\. Try to lower your batch size\\.\\\\nTry to allocate (.+?) tokens\\.\\\\n(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/allocation.py","lineNumber":224,"sourceCode":"\n    if is_dsv4:\n        bundle = out\n        out_cache_loc = None if bundle is None else bundle.out_full_loc\n        if batch is not None:\n            batch.out_cache_loc_dsv4 = bundle\n    else:\n        out_cache_loc = out\n\n    if out_cache_loc is None:\n        error_msg = (\n            f\"Prefill out of memory. Try to lower your batch size.\\n\"\n            f\"Try to allocate {extend_num_tokens} tokens.\\n\"\n            f\"{available_and_evictable_str(tree_cache)}\"\n        )\n        logger.error(error_msg)\n        if tree_cache is not None:\n            tree_cache.pretty_print()\n        raise RuntimeError(error_msg)\n\n    return out_cache_loc\n\n\ndef alloc_req_slots(\n    req_to_token_pool: ReqToTokenPool,\n    reqs: list[Req],\n    tree_cache: BasePrefixCache | None,\n) -> list[int]:\n    \"\"\"Allocate request slots from the pool.\n\n    Fail-loud: raises ``RuntimeError`` if the pool can't satisfy the batch. An\n    alloc failure here means the admission budget (``PrefillAdder``) was wrong\n    and should surface rather than be masked.\n    \"\"\"\n    num_reqs = len(reqs)\n    if isinstance(req_to_token_pool, HybridReqToTokenPool):\n        # Byte-coordinated for the shared allocator (accounts for the peer full","sourceCodeStart":206,"sourceCodeEnd":242,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/allocation.py#L206-L242","documentation":"The paged-KV extend allocation path (alloc_paged_token_slots_extend) failed to reserve extend_num_tokens slots for prefill growth even after evicting cached prefixes. This is the paged-cache variant of the generic token OOM, specific to extend/prefill batches (also used by NPU and reserve-extend wrappers).","triggerScenarios":"Prefilling a batch whose extend token count exceeds paged KV pool free + evictable pages; occurs on paged/hybrid allocators, NPU runs, and reserve-extend scheduling.","commonSituations":"Large chunked-prefill chunks (--chunked-prefill-size too big), many concurrent long prompts with radix cache pressure, or --mem-fraction-static too low for the paged pool.","solutions":["Lower --chunked-prefill-size and/or --max-running-requests","Raise --mem-fraction-static (more pages for the paged KV pool)","Reduce --context-length or prompt sizes","If prefix cache is full of evictable-but-fragmented pages, disable/flush radix caching to test"],"exampleFix":"# before\n--chunked-prefill-size 16384 --max-running-requests 128\n# after\n--chunked-prefill-size 4096 --max-running-requests 32","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    result = await client.generate(prompt)\nexcept Exception as e:\n    if \"Prefill out of memory\" in str(e):\n        prompt = chunk_or_truncate(prompt); retry_with_backoff()","preventionTips":["Set --chunked-prefill-size conservatively for long prompts","Cap concurrent long-prompt requests at the gateway","Track available-page metrics and throttle prefill admission"],"tags":["sglang","memory","paged-kv","prefill"],"backgroundTag":"kv-cache-out-of-memory","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}