sgl-project/sglang · error · RuntimeError

CUDA VMM recycler did not stop

Error message

CUDA VMM recycler did not stop

What it means

Raised during shutdown when the background recycler thread (which reclaims released VMM slices) fails to exit within the 1-second join timeout after _stop_recycler is set. A hung recycler usually means it is blocked in a driver call, a lock, or waiting on the publisher condition. Continuing would risk use-after-free of pool state, so shutdown aborts.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:626

        if self._allocation is None:
            return
        with torch.cuda.device(self.device_index):
            self._allocation.close()
        self._allocation = None

    def shutdown(self) -> None:
        with self._shutdown_lock:
            if self._closed:
                return
            with self._publisher_condition:
                self._closing = True
                while self._active_publishers:
                    self._publisher_condition.wait()

            self._stop_recycler.set()
            self._recycle_thread.join(timeout=1.0)
            if self._recycle_thread.is_alive():
                raise RuntimeError("CUDA VMM recycler did not stop")
            if self._fd_broker is not None:
                self._fd_broker.close()
                self._fd_broker = None
            self._release_allocation()
            self._closed = True


@dataclass
class _ImportedCudaVmmPool:
    pointer: int
    allocation_size: int
    memory: torch.Tensor | None

    def close(self) -> None:
        self.memory = None
        release_mappings(
            [
                (

View on GitHub (pinned to 0132848349)

Solutions

1. Retry shutdown after GPU work quiesces (ensure all publishers/requests finished first)
2. Increase the join timeout locally or drain in-flight transports before calling shutdown
3. If persistent, capture a py-spy dump of the recycle thread to find the blocking driver call / lock

# before shutdown: wait for publishers and recycle queue to drain
while transport._active_publishers:
    time.sleep(0.1)

try:
    transport.shutdown()
except RuntimeError as e:
    if "recycler did not stop" in str(e):
        log_thread_stacks()  # py-spy style dump
        transport.shutdown()  # retry once after drain

When it happens

Trigger: Calling shutdown() while the recycle thread is stuck in cuMemUnmap/release driver calls or holding the pool lock; heavy GPU load making the driver call exceed the 1s join timeout; deadlock with _active_publishers never draining.

Common situations: Shutdown under load or during CUDA context teardown; driver latency spikes; a publisher thread that never finished so the condition wait precedes a stuck state.

Related errors

• CUDA VMM multimodal pool is closing
• CUDA VMM POSIX FD broker failed
• CUDA VMM POSIX FD broker returned no file descriptor
• memory_size must be positive
• consumer_count must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1cc9ea369854166f. Report an issue: GitHub.