{"record":{"id":"d317b25cfb0662df","repo":"sgl-project/sglang","slug":"self-transport-name-pool-slot-generation-exhaust","errorCode":null,"errorMessage":"{self.transport_name} pool slot generation exhausted","messagePattern":"(.+?) pool slot generation exhausted","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/multimodal/transport/memory_pool.py","lineNumber":257,"sourceCode":"            return len(self._occupied)\n\n    def _allocate_locked(self, nbytes: int) -> Optional[PoolLease]:\n        allocation_bytes = align_up(nbytes, DATA_ALIGNMENT)\n        candidates = [\n            (end - start, index, start, end)\n            for index, (start, end) in enumerate(self._available_ranges)\n            if end - start >= allocation_bytes\n        ]\n        if not candidates or not self._available_slots:\n            return None\n        _, index, start, end = min(candidates)\n        self._available_ranges.pop(index)\n        if start + allocation_bytes < end:\n            self._available_ranges.append((start + allocation_bytes, end))\n        slot = self._available_slots.pop()\n        generation = self._slot_generations[slot] + 1\n        if generation > 0x7FFFFFFF:\n            raise RuntimeError(f\"{self.transport_name} pool slot generation exhausted\")\n        self._slot_generations[slot] = generation\n        ready_byte_offset = slot * self.control_words_per_slot * CONTROL_WORD_BYTES\n        lease = PoolLease(\n            start=start,\n            end=start + allocation_bytes,\n            nbytes=nbytes,\n            slot=slot,\n            generation=generation,\n            ready_byte_offset=ready_byte_offset,\n            ack_byte_offset=ready_byte_offset + CONTROL_WORD_BYTES,\n        )\n        self._occupied[slot] = lease\n        return lease\n\n    def _release_locked(self, lease: PoolLease) -> None:\n        active_lease = self._occupied.get(lease.slot)\n        if active_lease != lease:\n            raise RuntimeError(","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/multimodal/transport/memory_pool.py#L239-L275","documentation":"Each pool slot carries a 31-bit monotonically increasing generation counter used to detect stale leases (ABA protection). After 2^31-1 allocations of the same slot the counter would overflow, so the pool refuses further allocation.","triggerScenarios":"_allocate_locked (via copy_tensor) called ~2.1 billion times that reuse the same slot — practically only long-running servers with very high allocation churn or a pathological recycle loop.","commonSituations":"Ultra-long-lived serving process (weeks/months) recycling the same slot every iteration; stress tests that allocate/release in a tight loop.","solutions":["Restart the worker process when approaching the limit (operationally the cleanest fix)","Report upstream to sglang if genuinely hit — the counter width may need widening to 64-bit","Reduce allocation churn (larger batches, fewer copies) to slow generation growth"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    lease = pool.copy_tensor(t)\nexcept RuntimeError as e:\n    if 'generation exhausted' in str(e):\n        signal_worker_restart()  # operational fallback\n    else:\n        raise","preventionTips":["Treat this as a wear indicator: monitor allocation counts on long-lived workers","Recycle/restart workers periodically in multi-week serving deployments"],"tags":["multimodal","memory-pool","counter-overflow","long-running"],"backgroundTag":"resource-generation-overflow","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}