sgl-project/sglang · critical · RuntimeError

failed to move modules to {device}; rollback finished: error

Error message

failed to move modules to {device}; rollback finished: error={e}

What it means

RuntimeError raised by MemoryOccupationController._move_modules after it caught an exception while moving modules to a target device; it rolls back all already-moved modules to their source devices and then re-raises with the original error chained. The message means the move failed midway and state was restored.

Source

Thrown at python/sglang/multimodal_gen/runtime/managers/memory_managers/memory_occupation_controller.py:150

                module = modules[name]
                src_device_map[name] = _get_module_device(module)
                if device.startswith("cpu"):
                    _module_to_pinned_cpu(module)
                else:
                    module.to(device, non_blocking=True)
                moved.append(name)
                _move_unregistered_tensors(module, device)
            torch.cuda.synchronize()
        except Exception as e:
            logger.warning(
                f"[_move_modules] move failed, rollback started: target={device} moved={moved} error={e}",
            )
            for name in moved:
                module = modules.get(name)
                src_dev = src_device_map.get(name)
                module.to(src_dev)
                _move_unregistered_tensors(module, src_dev)
            raise RuntimeError(
                f"failed to move modules to {device}; rollback finished: error={e}"
            ) from e

    def _offload_active_modules_to_cpu(self) -> dict[str, str]:
        restore_map: dict[str, str] = {}
        for name, module in get_updatable_modules(self.pipeline).items():
            if _is_layerwise_offload_managed(module):
                continue
            device = _get_module_device(module)
            if not device.startswith("cpu"):
                restore_map[name] = device

        self._move_modules(list(restore_map.keys()), "cpu")
        self._clear_torch_device_cache()
        return restore_map

    def _restore_modules_to_original_devices(
        self, module_device_map: dict[str, str]

View on GitHub (pinned to 0132848349)

Solutions

  1. Free GPU memory (delete other processes/caches, torch.cuda.empty_cache()) and retry the restore
  2. Verify the target device string is valid and visible (cuda:0 within CUDA_VISIBLE_DEVICES)
  3. Check the chained 'error={e}' for the root cause and fix that (OOM, meta-device tensors, etc.)
  4. If using dtensor/FSDP modules, ensure _move_unregistered_tensors handles them or disable that path
Defensive patterns

Strategy: fallback

Validate before calling

free, _ = torch.cuda.mem_get_info()
required = estimate_model_bytes()
assert free > required, f"need ~{required/1e9:.1f}GB, only {free/1e9:.1f}GB free"

Try / catch

try:
    controller.resume_memory_occupation()
except RuntimeError as e:
    if "rollback finished" in str(e):
        logger.error("move failed, modules rolled back: %s", e.__cause__)
        torch.cuda.empty_cache()
        raise

Prevention

When it happens

Trigger: Calling offload (e.g. release_memory_occupation/_offload_active_modules_to_cpu) or restore while a module.to(device) throws — commonly CUDA OOM when restoring to GPU, invalid device id, or a tensor on an incompatible device/dtype.

Common situations: GPU out of memory when waking the model back up; wrong CUDA_VISIBLE_DEVICES mapping between offload and restore; mixed-precision/dt device ('dt' meta device) modules that can't be .to()'d to cuda; version changes in how dtensors are registered.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/174c7a7494d8e28b. Report an issue: GitHub.