huggingface/transformers · critical · RuntimeError

No requests can be scheduled and no requests can be offloade

Error message

No requests can be scheduled and no requests can be offloaded.

What it means

Raised in the ContinuousBatchingManager generation loop when scheduler.schedule_batch returns None (cache full) and the offloading manager cannot offload any more requests (returns 0). The loop that trades active requests for schedulability has hit a fixed point: nothing schedulable, nothing evictable — the system is deadlocked on memory.

Source

Thrown at src/transformers/generation/continuous_batching/continuous_api.py:387

        cancelled_states = self.scheduler.clear_cancelled_requests()
        # Also free CPU-offloaded cache for cancelled states. This is CPU-only, so it isn't batched like D2H transfers
        for state in cancelled_states:
            self.offloading_manager.free_request_cpu_cache(state)
        if not self.scheduler.has_pending_requests():
            return False

        # Schedule the next batch of requests
        requests_in_batch, use_decode_fast_path, num_q_tokens, max_kv_read = self.scheduler.schedule_batch(
            self.max_batch_tokens, self.cache.num_pages
        )

        # If requests_in_batch is None, it means the cache is full and no requests can be scheduled. We loop over active
        # requests and offload enough so that the remaining ones can all be scheduled. The loop is necessary because of
        # prefix sharing: offloading a fully shared request has 0 impact. Its termination is guaranteed.
        while requests_in_batch is None:
            # Stop case: no request can be offloaded.
            if self.offloading_manager.offload_requests() == 0:
                raise RuntimeError("No requests can be scheduled and no requests can be offloaded.")
            # Otherwise, the loop has offloaded at least one request, and we try scheduling again.
            requests_in_batch, use_decode_fast_path, num_q_tokens, max_kv_read = self.scheduler.schedule_batch(
                self.max_batch_tokens, self.cache.num_pages
            )

        # If requests_in_batch is an empty list, it means we have no requests to process anymore
        if not requests_in_batch:
            return False
        # If some active requests could not get new blocks, offload enough of them so it won't happen again next batch
        if self.scheduler.starved_requests:
            self.offloading_manager.offload_requests()  # NOTE: this only offload non-scheduled requests

        # Restore any CPU-offloaded requests that were just scheduled
        self.offloading_manager.restore_scheduled_requests(requests_in_batch)

        # Otherwise, we can continue with the non-empty batch and log in the dimensions before padding
        if logger.isEnabledFor(logging.DEBUG):
            logger.debug(

View on GitHub (pinned to a597f97485)

Solutions

1. Reduce concurrent load: fewer active requests / shorter max prompt lengths
2. Increase cache capacity (more memory, smaller cache dtype, smaller block overhead) so schedule_batch can fit at least one request
3. Increase cpu_offload_space (and install psutil) so offload_requests() has somewhere to evict to
4. Catch RuntimeError around the generation step and drain/restart the loop with a lower request limit

Example fix

# before
manager.add(request) for request in many_long_requests  # fill cache

# after
semaphore = asyncio.Semaphore(8)  # cap concurrent requests to what the cache can hold
async with semaphore:
    manager.add(request)

try:
    await manager.step()
except RuntimeError as e:
    if 'No requests can be scheduled' in str(e):
        await manager.stop()          # drain
        await asyncio.sleep(backoff)  # then re-open with fewer concurrent requests
        backoff = min(backoff * 2, 60)
    else:
        raise

When it happens

Trigger: All cache pages are held by requests that cannot be offloaded (CPU swap pool full / soft-reset ineligible) while new prefill also cannot fit; extremely small max_batch_tokens or num_pages combined with long prompts; offload pool sized 0 bytes.

Common situations: Tiny GPU with long-context requests filling every block; cpu_offload_space set too small; prefix-shared requests that offload to 0 effect until the pool is exhausted; concurrent request bursts exceeding capacity.

Related errors

• Failed to allocate {} blocks for request {}
• Memory footprint {} is more than available memory {}
• Invalid values: max_batch_tokens = {}, num_blocks = {}
• No real solution (discriminant = {})
• No positive solution (root = {})

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/f7ac41da0c389fdd. Report an issue: GitHub.