{"record":{"id":"f7ac41da0c389fdd","repo":"huggingface/transformers","slug":"no-requests-can-be-scheduled-and-no-requests-can-b","errorCode":null,"errorMessage":"No requests can be scheduled and no requests can be offloaded.","messagePattern":"No requests can be scheduled and no requests can be offloaded\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/transformers/generation/continuous_batching/continuous_api.py","lineNumber":387,"sourceCode":"        cancelled_states = self.scheduler.clear_cancelled_requests()\n        # Also free CPU-offloaded cache for cancelled states. This is CPU-only, so it isn't batched like D2H transfers\n        for state in cancelled_states:\n            self.offloading_manager.free_request_cpu_cache(state)\n        if not self.scheduler.has_pending_requests():\n            return False\n\n        # Schedule the next batch of requests\n        requests_in_batch, use_decode_fast_path, num_q_tokens, max_kv_read = self.scheduler.schedule_batch(\n            self.max_batch_tokens, self.cache.num_pages\n        )\n\n        # If requests_in_batch is None, it means the cache is full and no requests can be scheduled. We loop over active\n        # requests and offload enough so that the remaining ones can all be scheduled. The loop is necessary because of\n        # prefix sharing: offloading a fully shared request has 0 impact. Its termination is guaranteed.\n        while requests_in_batch is None:\n            # Stop case: no request can be offloaded.\n            if self.offloading_manager.offload_requests() == 0:\n                raise RuntimeError(\"No requests can be scheduled and no requests can be offloaded.\")\n            # Otherwise, the loop has offloaded at least one request, and we try scheduling again.\n            requests_in_batch, use_decode_fast_path, num_q_tokens, max_kv_read = self.scheduler.schedule_batch(\n                self.max_batch_tokens, self.cache.num_pages\n            )\n\n        # If requests_in_batch is an empty list, it means we have no requests to process anymore\n        if not requests_in_batch:\n            return False\n        # If some active requests could not get new blocks, offload enough of them so it won't happen again next batch\n        if self.scheduler.starved_requests:\n            self.offloading_manager.offload_requests()  # NOTE: this only offload non-scheduled requests\n\n        # Restore any CPU-offloaded requests that were just scheduled\n        self.offloading_manager.restore_scheduled_requests(requests_in_batch)\n\n        # Otherwise, we can continue with the non-empty batch and log in the dimensions before padding\n        if logger.isEnabledFor(logging.DEBUG):\n            logger.debug(","sourceCodeStart":369,"sourceCodeEnd":405,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/continuous_api.py#L369-L405","documentation":"Raised in the ContinuousBatchingManager generation loop when scheduler.schedule_batch returns None (cache full) and the offloading manager cannot offload any more requests (returns 0). The loop that trades active requests for schedulability has hit a fixed point: nothing schedulable, nothing evictable — the system is deadlocked on memory.","triggerScenarios":"All cache pages are held by requests that cannot be offloaded (CPU swap pool full / soft-reset ineligible) while new prefill also cannot fit; extremely small max_batch_tokens or num_pages combined with long prompts; offload pool sized 0 bytes.","commonSituations":"Tiny GPU with long-context requests filling every block; cpu_offload_space set too small; prefix-shared requests that offload to 0 effect until the pool is exhausted; concurrent request bursts exceeding capacity.","solutions":["Reduce concurrent load: fewer active requests / shorter max prompt lengths","Increase cache capacity (more memory, smaller cache dtype, smaller block overhead) so schedule_batch can fit at least one request","Increase cpu_offload_space (and install psutil) so offload_requests() has somewhere to evict to","Catch RuntimeError around the generation step and drain/restart the loop with a lower request limit"],"exampleFix":"# before\nmanager.add(request) for request in many_long_requests  # fill cache\n\n# after\nsemaphore = asyncio.Semaphore(8)  # cap concurrent requests to what the cache can hold\nasync with semaphore:\n    manager.add(request)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    await manager.step()\nexcept RuntimeError as e:\n    if 'No requests can be scheduled' in str(e):\n        await manager.stop()          # drain\n        await asyncio.sleep(backoff)  # then re-open with fewer concurrent requests\n        backoff = min(backoff * 2, 60)\n    else:\n        raise","preventionTips":["Cap concurrent requests with a semaphore sized to cache capacity","Size cpu_offload_space generously so offloading always has headroom","Monitor free cache pages and shed load before saturation"],"tags":["runtime","scheduling","memory","continuous-batching","deadlock"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}