{"record":{"id":"f6d9129263eaa3f4","repo":"sgl-project/sglang","slug":"cuda-vmm-multimodal-pool-failed","errorCode":null,"errorMessage":"CUDA VMM multimodal pool failed","messagePattern":"CUDA VMM multimodal pool failed","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/utils/cuda_vmm_transport_utils.py","lineNumber":340,"sourceCode":"        self._allocation = allocation\n        self.allocation_size = allocation_size\n        self.shareable_handle = (\n            bytes(exported.data) if self.use_fabric else int(exported)\n        )\n        logger.info(\n            \"CUDA VMM multimodal pool uses %s backing on device %d\",\n            allocation_handle_type_name(self.handle_type),\n            self.device_index,\n        )\n        self.memory_pool = memory_pool\n\n    @property\n    def control_size(self) -> int:\n        return align_up(self.consumer_count * _CONTROL_WORD_BYTES, _CONTROL_ALIGNMENT)\n\n    def _raise_if_failed(self) -> None:\n        if self._pool_error is not None:\n            raise RuntimeError(\"CUDA VMM multimodal pool failed\") from self._pool_error\n        if self._fd_broker is not None:\n            self._fd_broker.raise_if_failed()\n\n    def _reserve_for_publish(self, required_size: int) -> _CudaVmmMemoryChunk | None:\n        with self._publisher_condition:\n            self._raise_if_failed()\n            if self._closing or self._closed:\n                raise RuntimeError(\"CUDA VMM multimodal pool is closing\")\n            chunk = self._reserve_chunk(required_size)\n            if chunk is not None:\n                self._active_publishers += 1\n            return chunk\n\n    def _finish_publish(self) -> None:\n        with self._publisher_condition:\n            self._active_publishers -= 1\n            if self._active_publishers == 0:\n                self._publisher_condition.notify_all()","sourceCodeStart":322,"sourceCodeEnd":358,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_transport_utils.py#L322-L358","documentation":"Central health check for the VMM pool: if any earlier pool operation (allocation, mapping, broker) stored _pool_error, subsequent wrap_tensor/wrap_tensors/_reserve_for_publish re-raise it wrapped as this RuntimeError. The chained cause holds the actual failure.","triggerScenarios":"Any prior asynchronous/background allocation or broker error latching _pool_error; then any later call into wrap_tensors triggers the raise.","commonSituations":"OOM during pool allocation, driver errors during physical mapping, or broker failure (6371) surfacing lazily at the next multimodal publish.","solutions":["Inspect __cause__ for the first failure — fix that, not this wrapper","Free GPU memory / shrink memory_size if it was an allocation failure","Restart the process after fixing the root cause; the pool is permanently failed once latched","Check driver/fabric support if mapping failed"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    wrapped = pool.wrap_tensors(ts)\nexcept RuntimeError as e:\n    if \"pool failed\" in str(e):\n        report_root_cause(e.__cause__); restart_worker()","preventionTips":["Monitor the first allocation failure; later errors are echoes","Size memory_size to fit expected load to avoid latched OOM","Treat a latched pool error as fatal for the process"],"tags":["cuda","vmm","pool","lazy-failure","runtimeerror"],"backgroundTag":"resource-pool-exhausted-or-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}