sgl-project/sglang · critical · RuntimeError

Out of memory. Try to lower your batch size.\nTry to allocat

Error message

Out of memory. Try to lower your batch size.\nTry to allocate {num_tokens} tokens.\n{available_and_evictable_str(tree_cache)}

What it means

alloc_token_slots could not find num_tokens free KV-cache slots even after considering evictable prefix-cache entries, so the request cannot be admitted. The message reports how many tokens were needed and how much memory (including evictable cached prefixes) was available.

Source

Thrown at python/sglang/srt/mem_cache/allocation.py:168

def alloc_token_slots(
    tree_cache: BasePrefixCache,
    num_tokens: int,
):
    allocator = tree_cache.token_to_kv_pool_allocator
    evict_from_tree_cache(tree_cache, num_tokens)

    out_cache_loc = allocator.alloc(num_tokens)

    if out_cache_loc is None:
        error_msg = (
            f"Out of memory. Try to lower your batch size.\n"
            f"Try to allocate {num_tokens} tokens.\n"
            f"{available_and_evictable_str(tree_cache)}"
        )
        logger.error(error_msg)
        if tree_cache is not None:
            tree_cache.pretty_print()
        raise RuntimeError(error_msg)

    return out_cache_loc


def alloc_paged_token_slots_extend(
    tree_cache: BasePrefixCache,
    prefix_lens: torch.Tensor,
    prefix_lens_cpu: torch.Tensor,
    seq_lens: torch.Tensor,
    seq_lens_cpu: torch.Tensor,
    last_loc: torch.Tensor,
    extend_num_tokens: int,
    req_pool_indices: Optional[torch.Tensor] = None,
    batch=None,
):
    # Over estimate the number of tokens: assume each request needs a new page.
    allocator = tree_cache.token_to_kv_pool_allocator
    num_tokens = extend_num_tokens + len(seq_lens_cpu) * allocator.page_size

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce --max-running-requests or batch size
2. Lower individual request length / --context-length
3. Increase --mem-fraction-static so the KV pool is larger
4. Clear or shrink the prefix cache (--chunked-prefill-size tuning, or disable prefix caching to avoid evictable-vs-usable confusion)
5. Restart with a smaller model or tensor parallelism to free per-GPU KV memory

Example fix

# before
python -m sglang.launch_server --model meta-llama/Llama-3-70B --max-running-requests 256
# after
python -m sglang.launch_server --model meta-llama/Llama-3-70B --max-running-requests 64 --mem-fraction-static 0.9

try:
    out = engine.generate(prompt, sampling_params)
except RuntimeError as e:
    if "Out of memory" in str(e):
        reduce_batch_and_retry()
    else:
        raise

When it happens

Trigger: A batch's token allocation (extend/decode/spec-decode, or KV-reuse paths) exceeds token_to_kv_pool available + evictable size — typically very long prompts, huge batch sizes, or a small --mem-fraction-static leaving a tiny KV pool.

Common situations: Long-context requests with aggressive prefix caching filling the pool; max-running-requests too high; context length close to KV capacity; memory squeezed by a large model or HiCache/reserved buffers.

Related errors

• v_cache must be provided
• k_cache can only be None when only_qv=True
• Invalid mode: {mode}, must be one of 'write', 'read', 'skip'
• MXFP8 KV cache requires the FA4 backend.
• Prefill out of memory. Try to lower your batch size.\nTry to

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4c7e8eb4d955c1d3. Report an issue: GitHub.