sgl-project/sglang · error · RuntimeError

VmmReservation.map_existing after close

Error message

VmmReservation.map_existing after close

What it means

VmmReservation.map_existing refuses to map a caller-owned physical allocation (via cuMemMap of an exported handle) after the reservation has been closed, because the underlying VA range has already been released to the driver.

Solutions

1. Audit cleanup paths to ensure no code path maps after close (guard with try/finally ordering).
2. Check reservation._closed / use a wrapper that raises a clear domain error before calling.
3. Create a fresh reservation and redo the import instead of reusing a closed one.

Example fix

// before
res.map_existing(offset, size, handle)  # res already closed

// after
if res._closed:
    res = VmmReservation.reserve(total_size)
res.map_existing(offset, size, handle)

def is_open(res) -> bool:
    return not getattr(res, "_closed", True)

try:
    res.map_existing(offset, size, handle)
except RuntimeError as e:
    if "after close" in str(e):
        res = make_fresh_reservation(total_size)
        res.map_existing(offset, size, handle)
    else:
        raise

When it happens

Trigger: Calling map_existing on a VmmReservation whose close() has already run (the _closed flag is set); typically when mapping peer views or setup-layer handles after teardown began.

Common situations: Error-path cleanup ordering: an exception triggers close() of the reservation, then a subsequent retry or teardown handler calls map_existing; or holding a stale reservation object across a model reload / engine shutdown.

Related errors

• CUDA VMM multimodal pool is closing
• CUDA VMM tensor has already released its pool slice
• A CUDA VMM-enabled model must provide a multimodal processor
• attention group range
• consumer_count must be 1, the attention TP size, or the…

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c38b3ff8a49f5549. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/utils/cuda_vmm_utils.py:596

            if handle is not None:
                try:
                    check_drv(drv.cuMemRelease(handle), "cuMemRelease(local rollback)")
                except BaseException as cleanup_error:
                    cleanup_errors.append(cleanup_error)
            if cleanup_errors:
                error.add_note(
                    f"{len(cleanup_errors)} CUDA VMM rollback operation(s) also failed"
                )
                raise error from cleanup_errors[0]
            raise

        self._mappings.append((address, size, handle))
        return handle

    def map_existing(self, offset: int, size: int, handle) -> None:
        """Map a caller-owned physical allocation into this reservation."""
        if self._closed:
            raise RuntimeError("VmmReservation.map_existing after close")
        offset, size = int(offset), int(size)
        drv = _get_cuda_driver()
        address = self.base + offset
        mapped = False
        try:
            check_drv(
                drv.cuMemMap(address, size, 0, handle, 0),
                "cuMemMap(existing)",
            )
            mapped = True
            check_drv(
                drv.cuMemSetAccess(
                    address,
                    size,
                    self._access_descs,
                    len(self._access_descs),
                ),
                "cuMemSetAccess(existing)",

View on GitHub (pinned to 0132848349)