What causes "CUDA VMM multimodal pool is closing" in sgl-project/sglang?

Drain/await in-flight publishers before calling close() on the pool

sgl-project/sglang · error · RuntimeError

CUDA VMM multimodal pool is closing

Error message

CUDA VMM multimodal pool is closing

What it means

A publisher called _reserve_for_publish after close() began (self._closing/_closed set). The pool refuses new publish reservations during shutdown to avoid allocating into memory being torn down.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:348

            self.device_index,
        )
        self.memory_pool = memory_pool

    @property
    def control_size(self) -> int:
        return align_up(self.consumer_count * _CONTROL_WORD_BYTES, _CONTROL_ALIGNMENT)

    def _raise_if_failed(self) -> None:
        if self._pool_error is not None:
            raise RuntimeError("CUDA VMM multimodal pool failed") from self._pool_error
        if self._fd_broker is not None:
            self._fd_broker.raise_if_failed()

    def _reserve_for_publish(self, required_size: int) -> _CudaVmmMemoryChunk | None:
        with self._publisher_condition:
            self._raise_if_failed()
            if self._closing or self._closed:
                raise RuntimeError("CUDA VMM multimodal pool is closing")
            chunk = self._reserve_chunk(required_size)
            if chunk is not None:
                self._active_publishers += 1
            return chunk

    def _finish_publish(self) -> None:
        with self._publisher_condition:
            self._active_publishers -= 1
            if self._active_publishers == 0:
                self._publisher_condition.notify_all()

    def wrap_tensor(self, tensor: torch.Tensor):
        self._raise_if_failed()
        if not tensor.is_contiguous():
            tensor = tensor.contiguous()
        data_nbytes = tensor.numel() * tensor.element_size()
        required_size = align_up(self.control_size + data_nbytes, _CONTROL_ALIGNMENT)
        source_bytes = tensor.reshape(-1).view(torch.uint8)

View on GitHub (pinned to 0132848349)

Solutions

Drain/await in-flight publishers before calling close() on the pool
Catch RuntimeError in publish paths during shutdown and treat as a clean cancel
Ensure worker threads observe the shutdown flag before publishing

Example fix

# before
pool.close()  # while worker still publishing
# after
await drain_publishers()
pool.close()

Defensive patterns

Strategy: try-catch

Try / catch

try:
    chunk = pool_reserve(publish_fn)
except RuntimeError as e:
    if "closing" in str(e):
        return  # clean cancel during shutdown

Prevention

Drain in-flight publishers before close()
Set shutdown flags workers check before publishing
Catch and swallow 'is closing' during graceful shutdown paths

When it happens

Trigger: Race between one thread shutting down the pool (server shutdown, scheduler exit) and another thread wrapping tensors for publish — e.g. an in-flight multimodal batch finishing during shutdown.

Common situations: Unclean shutdowns with in-flight requests; signal handling that closes the transport while workers still publish; missing drain of publishers before close.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b30aa8e9e7887349. Report an issue: GitHub.