{"record":{"id":"75399c058b7579ff","repo":"huggingface/transformers","slug":"failed-to-allocate-blocks-for-request","errorCode":null,"errorMessage":"Failed to allocate {} blocks for request {}","messagePattern":"Failed to allocate (.+?) blocks for request (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"src/transformers/generation/continuous_batching/cache.py","lineNumber":335,"sourceCode":"        \"\"\"Returns a boolean indicating if the allocation of (num_requested_blocks) blocks will be successful.\"\"\"\n        return self.blocks_needed(num_requested_blocks, allocated_blocks) <= self.get_num_free_blocks()\n\n    def blocks_in_use(self, request_id: str) -> int:\n        \"\"\"Returns the total number of physical blocks currently referenced by a request across all layer groups.\"\"\"\n        return sum(len(cm.block_table.get(request_id, ())) for cm in self.group_cache_managers)\n\n    def allocate_blocks(self, n_blocks: int, request_id: str, allocated_blocks: int) -> int | None:\n        \"\"\"Allocate cache blocks across all layer groups for a given request. Actual allocation is done by the cache\n        managers, and this method only returns the maximum number of blocks actually allocated across all managers.\"\"\"\n        # First check allocation will be successful before starting, to avoid partial allocations\n        if not self.will_allocation_be_successful(n_blocks, allocated_blocks):\n            return None\n        # Allocate blocks across all cache managers\n        max_allocated = 0\n        for cm in self.group_cache_managers:\n            num_allocated_blocks = cm.allocate_blocks(n_blocks, request_id, self._block_manager)\n            if num_allocated_blocks is None:\n                raise ValueError(f\"Failed to allocate {n_blocks} blocks for request {request_id}\")\n            max_allocated = max(max_allocated, num_allocated_blocks)\n        return max_allocated\n\n    def free_blocks(self, request_id: str) -> None:\n        \"\"\"Free all allocated cache blocks for a given request across all layer groups. Actual deallocation is done\n        by the cache managers.\"\"\"\n        for cm in self.group_cache_managers:\n            cm.free_blocks(request_id, self._block_manager)\n\n    def get_num_free_blocks(self) -> int:\n        \"\"\"Get the current number of unallocated blocks available for new requests.\"\"\"\n        return self._block_manager.num_free_blocks\n\n    def extend_read_and_write_indices(\n        self,\n        request_id: str,\n        past_length: int,\n        query_length: int,","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/cache.py#L317-L353","documentation":"Internal ValueError from Cache.allocate_blocks(): a pre-check said the allocation should succeed, but an individual group cache manager returned None from allocate_blocks, meaning it could not actually reserve the blocks. This signals a bookkeeping inconsistency between the block manager and group allocators (or a race in concurrent allocation), not normal out-of-memory — OOM is normally handled by the will_allocation_be_successful() pre-check returning False.","triggerScenarios":"Concurrent requests racing for the last free blocks across multiple threads/processes; bugs in custom block managers; mismatches between group allocators' capacity accounting after prefix sharing frees.","commonSituations":"Running the continuous-batching server under heavy concurrency; custom forks of the cache that alter free/allocate logic.","solutions":["If you are a user (not modifying internals): report the issue on the transformers GitHub with a reproducer — this is an internal invariant violation","Reduce concurrency / max running requests so allocations never approach the block limit","Give the cache more memory (raise max_memory_percent or num_blocks) so the race window closes","If you patched the cache, audit that every allocate/free path updates both BlockManager and group allocators consistently"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if cache.get_num_free_blocks() < blocks_needed:\n    # queue the request instead of allocating\n    enqueue(request)","typeGuard":null,"tryCatchPattern":"try:\n    cache.allocate_blocks(n_blocks, request_id, allocated)\nexcept ValueError as e:\n    if 'Failed to allocate' in str(e):\n        # internal invariant break: dump state and report upstream\n        log_state(cache, request_id)\n        raise RuntimeError('cache allocator desynchronized; see logs') from e\n    raise","preventionTips":["Cap concurrent requests below the block budget so allocation never races the limit","Report reproductions to the transformers maintainers — this is an internal bug path","Monitor get_num_free_blocks() and shed load before it hits zero"],"tags":["continuous-batching","kv-cache","internal-invariant","concurrency","memory"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}