{"record":{"id":"edc4df790b100979","repo":"sgl-project/sglang","slug":"cuda-vmm-pool-has-no-occupied-slice-at-control-off","errorCode":null,"errorMessage":"CUDA VMM pool has no occupied slice at control offset {control_offset}","messagePattern":"CUDA VMM pool has no occupied slice at control offset (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/cuda_vmm_transport_utils.py","lineNumber":535,"sourceCode":"\n    def _release_reserved_chunk(self, chunk: _CudaVmmMemoryChunk) -> None:\n        if chunk in self.occupied_chunks:\n            self.occupied_chunks.remove(chunk)\n        self.available_chunks.append(_CudaVmmMemoryChunk(chunk.start, chunk.end))\n        self._merge_chunks()\n\n    def _cancel_control_offset(self, control_offset: int) -> None:\n        with self._lock:\n            chunk = next(\n                (\n                    chunk\n                    for chunk in self.occupied_chunks\n                    if chunk.start == control_offset\n                ),\n                None,\n            )\n            if chunk is None:\n                raise RuntimeError(\n                    \"CUDA VMM pool has no occupied slice at control offset \"\n                    f\"{control_offset}\"\n                )\n            self._release_reserved_chunk(chunk)\n\n    def cancel_proxy(self, proxy: CudaVmmTensorTransportProxy) -> None:\n        \"\"\"Return a published slice when its request was never dispatched.\"\"\"\n        if isinstance(proxy, CudaVmmPackedTensorTransportProxy):\n            proxy._packed_owner.cancel_from_pool(self)\n            return\n        self._cancel_control_offset(proxy.control_offset)\n\n    def _warn_pool_full_once(self, data_nbytes: int) -> None:\n        if self._pool_full_warned:\n            return\n        self._pool_full_warned = True\n        logger.warning(\n            \"CUDA VMM multimodal pool has no free chunk for a %.2f MiB tensor \"","sourceCodeStart":517,"sourceCodeEnd":553,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/cuda_vmm_transport_utils.py#L517-L553","documentation":"Raised when cancel_proxy/cancel_from_pool is asked to release a reserved pool slice at a given control offset, but no occupied chunk in the CUDA VMM memory pool starts at that offset. This means the slice was already released, was never allocated, or the offset is stale/corrupted. It signals an accounting mismatch between the transport proxy bookkeeping and the pool's occupied chunk list.","triggerScenarios":"Calling cancel_proxy() twice for the same proxy; using a CudaVmmTensorTransportProxy after its chunk was already released via acknowledge_consumption(); passing a control_offset that was never returned by the pool's reserve path.","commonSituations":"Double-cancel after a partial dispatch failure where some slices were already released; a request retry path that re-cancels items; racing shutdown() while cancels are in flight.","solutions":["Ensure cancel is called exactly once per slice (guard with a released flag or set the field to None immediately, as the finally block does)","Check chunk state / pool occupancy before cancelling: query whether an occupied chunk exists at the offset","Audit error paths in prepare_for_dispatch / _send_one_request for double-release after cancellation"],"exampleFix":"// before\npool.cancel_proxy(proxy)\n\n// after\nif proxy.released:\n    return\npool.cancel_proxy(proxy)\nproxy.released = True","handlingStrategy":"validation","validationCode":"occupied = [c.start for c in pool.occupied_chunks]\nif control_offset not in occupied:\n    logger.warning(\"slice already released at %d\", control_offset)\n    return","typeGuard":null,"tryCatchPattern":"try:\n    pool.cancel_proxy(proxy)\nexcept RuntimeError as e:\n    if \"no occupied slice\" in str(e):\n        return  # idempotent cancel\n    raise","preventionTips":["Cancel each slice exactly once; null the field immediately after cancelling","Keep proxy release state on the proxy object and check it before cancel"],"tags":["cuda","vmm","double-free","memory-pool"],"backgroundTag":"double-release-resource","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}