{"record":{"id":"1cc9ea369854166f","repo":"sgl-project/sglang","slug":"cuda-vmm-recycler-did-not-stop","errorCode":null,"errorMessage":"CUDA VMM recycler did not stop","messagePattern":"CUDA VMM recycler did not stop","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_transport_utils.py","lineNumber":626,"sourceCode":"        if self._allocation is None:\n            return\n        with torch.cuda.device(self.device_index):\n            self._allocation.close()\n        self._allocation = None\n\n    def shutdown(self) -> None:\n        with self._shutdown_lock:\n            if self._closed:\n                return\n            with self._publisher_condition:\n                self._closing = True\n                while self._active_publishers:\n                    self._publisher_condition.wait()\n\n            self._stop_recycler.set()\n            self._recycle_thread.join(timeout=1.0)\n            if self._recycle_thread.is_alive():\n                raise RuntimeError(\"CUDA VMM recycler did not stop\")\n            if self._fd_broker is not None:\n                self._fd_broker.close()\n                self._fd_broker = None\n            self._release_allocation()\n            self._closed = True\n\n\n@dataclass\nclass _ImportedCudaVmmPool:\n    pointer: int\n    allocation_size: int\n    memory: torch.Tensor | None\n\n    def close(self) -> None:\n        self.memory = None\n        release_mappings(\n            [\n                (","sourceCodeStart":608,"sourceCodeEnd":644,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_transport_utils.py#L608-L644","documentation":"Raised during shutdown when the background recycler thread (which reclaims released VMM slices) fails to exit within the 1-second join timeout after _stop_recycler is set. A hung recycler usually means it is blocked in a driver call, a lock, or waiting on the publisher condition. Continuing would risk use-after-free of pool state, so shutdown aborts.","triggerScenarios":"Calling shutdown() while the recycle thread is stuck in cuMemUnmap/release driver calls or holding the pool lock; heavy GPU load making the driver call exceed the 1s join timeout; deadlock with _active_publishers never draining.","commonSituations":"Shutdown under load or during CUDA context teardown; driver latency spikes; a publisher thread that never finished so the condition wait precedes a stuck state.","solutions":["Retry shutdown after GPU work quiesces (ensure all publishers/requests finished first)","Increase the join timeout locally or drain in-flight transports before calling shutdown","If persistent, capture a py-spy dump of the recycle thread to find the blocking driver call / lock"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# before shutdown: wait for publishers and recycle queue to drain\nwhile transport._active_publishers:\n    time.sleep(0.1)","typeGuard":null,"tryCatchPattern":"try:\n    transport.shutdown()\nexcept RuntimeError as e:\n    if \"recycler did not stop\" in str(e):\n        log_thread_stacks()  # py-spy style dump\n        transport.shutdown()  # retry once after drain","preventionTips":["Quiesce all in-flight requests/publishers before shutdown","Monitor recycler thread health during long runs"],"tags":["cuda","vmm","shutdown","threading","timeout"],"backgroundTag":"thread-join-timeout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}