sgl-project/sglang · error · RuntimeError

CUDA VMM tensor has already released its pool slice

Error message

CUDA VMM tensor has already released its pool slice

What it means

Raised when reconstruct_on_target_device is called after the tensor already acknowledged consumption, which releases its slice back to the shared VMM pool. Once released, the underlying memory may be reused by other producers, so reconstruction is refused to prevent reading aliased data.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:815

    def acknowledge_consumption(self, consumer_count: int | None = None) -> None:
        consumer_count = self._resolve_consumer_count(consumer_count)
        device_index = torch.cuda.current_device()
        with torch.cuda.device(device_index):
            self._acknowledge_consumption(device_index, consumer_count)

    def reconstruct_on_target_device(
        self, rebuild_device_idx, consumer_count: int | None = None
    ):
        consumer_count = self._resolve_consumer_count(consumer_count)
        rebuild_device = torch.device(f"cuda:{rebuild_device_idx}")
        if (
            isinstance(self.reconstruct_tensor, torch.Tensor)
            and self.reconstruct_tensor.device == rebuild_device
        ):
            return self.reconstruct_tensor
        if self._consumer_acknowledged:
            raise RuntimeError("CUDA VMM tensor has already released its pool slice")

        pool = self._pool(rebuild_device_idx)
        try:
            with torch.cuda.device(rebuild_device):
                source = pool.memory[
                    self.data_offset : self.data_offset + self.data_nbytes
                ]
                reconstructed = torch.empty(
                    self.shape, dtype=self.dtype, device=rebuild_device
                ).contiguous()
                reconstructed.reshape(-1).view(torch.uint8).copy_(
                    source, non_blocking=True
                )
                self._acknowledge_consumption(rebuild_device_idx, consumer_count)
        except BaseException as error:
            try:
                with torch.cuda.device(rebuild_device):
                    self._acknowledge_consumption(rebuild_device_idx, consumer_count)

View on GitHub (pinned to 0132848349)

Solutions

1. Reconstruct the tensor first, then acknowledge (call reconstruct_on_target_device before acknowledge_consumption)
2. Cache the reconstructed tensor — the method returns the cached tensor if device matches, so keep and reuse it
3. Remove duplicate reconstruction attempts after release

Example fix

// before
proxy.acknowledge_consumption()
t = proxy.reconstruct_on_target_device(idx)

// after
t = proxy.reconstruct_on_target_device(idx)
proxy.acknowledge_consumption()

if proxy._consumer_acknowledged:
    return cached  # do not reconstruct after release

def can_reconstruct(proxy) -> bool:
    return not proxy._consumer_acknowledged

When it happens

Trigger: Calling reconstruct_on_target_device() after acknowledge_consumption() on the same proxy; a consumer acknowledging early then lazily trying to materialize the tensor.

Common situations: Eager-release optimizations that acknowledge before the tensor is actually used; reorderings where acknowledgement happens in a prefetch path but reconstruction happens later.

Related errors

• CUDA VMM multimodal pool is closing
• VmmReservation.map_existing after close
• CUDA VMM POSIX FD broker failed
• CUDA VMM POSIX FD broker returned no file descriptor
• memory_size must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/51eb1be402967412. Report an issue: GitHub.