sgl-project/sglang · critical · RuntimeError

Prefill out of memory. Try to lower your batch size.\nTry to

Error message

Prefill out of memory. Try to lower your batch size.\nTry to allocate {extend_num_tokens} tokens.\n{available_and_evictable_str(tree_cache)}

What it means

The paged-KV extend allocation path (alloc_paged_token_slots_extend) failed to reserve extend_num_tokens slots for prefill growth even after evicting cached prefixes. This is the paged-cache variant of the generic token OOM, specific to extend/prefill batches (also used by NPU and reserve-extend wrappers).

Source

Thrown at python/sglang/srt/mem_cache/allocation.py:224

    if is_dsv4:
        bundle = out
        out_cache_loc = None if bundle is None else bundle.out_full_loc
        if batch is not None:
            batch.out_cache_loc_dsv4 = bundle
    else:
        out_cache_loc = out

    if out_cache_loc is None:
        error_msg = (
            f"Prefill out of memory. Try to lower your batch size.\n"
            f"Try to allocate {extend_num_tokens} tokens.\n"
            f"{available_and_evictable_str(tree_cache)}"
        )
        logger.error(error_msg)
        if tree_cache is not None:
            tree_cache.pretty_print()
        raise RuntimeError(error_msg)

    return out_cache_loc


def alloc_req_slots(
    req_to_token_pool: ReqToTokenPool,
    reqs: list[Req],
    tree_cache: BasePrefixCache | None,
) -> list[int]:
    """Allocate request slots from the pool.

    Fail-loud: raises ``RuntimeError`` if the pool can't satisfy the batch. An
    alloc failure here means the admission budget (``PrefillAdder``) was wrong
    and should surface rather than be masked.
    """
    num_reqs = len(reqs)
    if isinstance(req_to_token_pool, HybridReqToTokenPool):
        # Byte-coordinated for the shared allocator (accounts for the peer full

View on GitHub (pinned to 0132848349)

Solutions

1. Lower --chunked-prefill-size and/or --max-running-requests
2. Raise --mem-fraction-static (more pages for the paged KV pool)
3. Reduce --context-length or prompt sizes
4. If prefix cache is full of evictable-but-fragmented pages, disable/flush radix caching to test

Example fix

# before
--chunked-prefill-size 16384 --max-running-requests 128
# after
--chunked-prefill-size 4096 --max-running-requests 32

try:
    result = await client.generate(prompt)
except Exception as e:
    if "Prefill out of memory" in str(e):
        prompt = chunk_or_truncate(prompt); retry_with_backoff()

When it happens

Trigger: Prefilling a batch whose extend token count exceeds paged KV pool free + evictable pages; occurs on paged/hybrid allocators, NPU runs, and reserve-extend scheduling.

Common situations: Large chunked-prefill chunks (--chunked-prefill-size too big), many concurrent long prompts with radix cache pressure, or --mem-fraction-static too low for the paged pool.

Related errors

• Decode out of memory. Try to lower your batch size.\nTry to
• CuteDSL MLA backend is only supported on Blackwell GPUs (SM1
• DSA indexer weights_proj LoRA is incompatible with piecewise
• FlashInferKDAKernel has no prefill kernel; keep prefill on T
• Out of memory. Try to lower your batch size.\nTry to allocat

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0aa273818a8c7e29. Report an issue: GitHub.