sgl-project/sglang · critical · RuntimeError

Decode out of memory. Try to lower your batch size.\nTry to

Error message

Decode out of memory. Try to lower your batch size.\nTry to allocate {len(seq_lens) * token_per_req} tokens.\n{available_and_evictable_str(tree_cache)}

What it means

The paged-KV decode allocation (alloc_paged_token_slots_decode) cannot allocate len(seq_lens) * token_per_req slots (one new page-slot per sequence per decode step, times tokens per request for spec-decode/multi-token decode). All running sequences must grow their KV every step, so a full pool aborts the batch.

Source

Thrown at python/sglang/srt/mem_cache/allocation.py:523

    if is_dsv4:
        bundle = out
        out_cache_loc = None if bundle is None else bundle.out_full_loc
        if batch is not None:
            batch.out_cache_loc_dsv4 = bundle
    else:
        out_cache_loc = out

    if out_cache_loc is None:
        error_msg = (
            f"Decode out of memory. Try to lower your batch size.\n"
            f"Try to allocate {len(seq_lens) * token_per_req} tokens.\n"
            f"{available_and_evictable_str(tree_cache)}"
        )
        logger.error(error_msg)
        if tree_cache is not None:
            tree_cache.pretty_print()
        raise RuntimeError(error_msg)

    return out_cache_loc


def alloc_for_decode(batch: ScheduleBatch, token_per_req: int) -> torch.Tensor:
    """
    Allocate KV cache for decode batch and write to req_to_token_pool.

    Returns:
        out_cache_loc: allocated cache locations
    """

    batch.maybe_evict_swa()

    seq_lens_gpu = batch.seq_lens
    bs = seq_lens_gpu.shape[0]

    if _alloc_page_size(batch) == 1:

View on GitHub (pinned to 0132848349)

Solutions

1. Lower --max-running-requests (directly shrinks len(seq_lens))
2. Increase --mem-fraction-static
3. If using speculative decoding, reduce num speculative tokens or disable it
4. Reduce --context-length so fewer pages are held per sequence

Example fix

# before
--max-running-requests 512 --speculative-num-steps 5
# after
--max-running-requests 128 --speculative-num-steps 3

try:
    token = await client.generate(prompt, sampling_params, stream=True)
except RuntimeError as e:
    if "Decode out of memory" in str(e):
        await asyncio.sleep(backoff); retry()  # scheduler retries after retraction
    raise

When it happens

Trigger: Decode-step allocation on a paged KV cache when running batch size × token_per_req exceeds free + evictable pages; aggravated by speculative decoding where token_per_req > 1, or by very large running batches.

Common situations: Large --max-running-requests during long decode phases filling pages; spec-decode draft tokens multiplying per-step allocation; small mem-fraction leaving too few pages; SWA/hybrid models fragmenting pages.

Related errors

• Prefill out of memory. Try to lower your batch size.\nTry to
• FlashKDAKernel only supports prefill (extend)
• NvidiaKDAKernel is prefill-only
• PtxKDAKernel is prefill-only
• Out of memory. Try to lower your batch size.\nTry to allocat

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/452006b5ad686a05. Report an issue: GitHub.