sgl-project/sglang · critical · RuntimeError

CUDA VMM multimodal pool failed

Error message

CUDA VMM multimodal pool failed

What it means

Central health check for the VMM pool: if any earlier pool operation (allocation, mapping, broker) stored _pool_error, subsequent wrap_tensor/wrap_tensors/_reserve_for_publish re-raise it wrapped as this RuntimeError. The chained cause holds the actual failure.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:340

        self._allocation = allocation
        self.allocation_size = allocation_size
        self.shareable_handle = (
            bytes(exported.data) if self.use_fabric else int(exported)
        )
        logger.info(
            "CUDA VMM multimodal pool uses %s backing on device %d",
            allocation_handle_type_name(self.handle_type),
            self.device_index,
        )
        self.memory_pool = memory_pool

    @property
    def control_size(self) -> int:
        return align_up(self.consumer_count * _CONTROL_WORD_BYTES, _CONTROL_ALIGNMENT)

    def _raise_if_failed(self) -> None:
        if self._pool_error is not None:
            raise RuntimeError("CUDA VMM multimodal pool failed") from self._pool_error
        if self._fd_broker is not None:
            self._fd_broker.raise_if_failed()

    def _reserve_for_publish(self, required_size: int) -> _CudaVmmMemoryChunk | None:
        with self._publisher_condition:
            self._raise_if_failed()
            if self._closing or self._closed:
                raise RuntimeError("CUDA VMM multimodal pool is closing")
            chunk = self._reserve_chunk(required_size)
            if chunk is not None:
                self._active_publishers += 1
            return chunk

    def _finish_publish(self) -> None:
        with self._publisher_condition:
            self._active_publishers -= 1
            if self._active_publishers == 0:
                self._publisher_condition.notify_all()

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect __cause__ for the first failure — fix that, not this wrapper
2. Free GPU memory / shrink memory_size if it was an allocation failure
3. Restart the process after fixing the root cause; the pool is permanently failed once latched
4. Check driver/fabric support if mapping failed

try:
    wrapped = pool.wrap_tensors(ts)
except RuntimeError as e:
    if "pool failed" in str(e):
        report_root_cause(e.__cause__); restart_worker()

When it happens

Trigger: Any prior asynchronous/background allocation or broker error latching _pool_error; then any later call into wrap_tensors triggers the raise.

Common situations: OOM during pool allocation, driver errors during physical mapping, or broker failure (6371) surfacing lazily at the next multimodal publish.

Related errors

• CUDA VMM multimodal transport selected POSIX_FD, but this po
• CUDA VMM POSIX FD broker failed
• CUDA VMM POSIX FD broker returned no file descriptor
• memory_size must be positive
• consumer_count must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f6d9129263eaa3f4. Report an issue: GitHub.