{"record":{"id":"174c7a7494d8e28b","repo":"sgl-project/sglang","slug":"failed-to-move-modules-to-device-rollback-finis","errorCode":null,"errorMessage":"failed to move modules to {device}; rollback finished: error={e}","messagePattern":"failed to move modules to (.+?); rollback finished: error=(.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/managers/memory_managers/memory_occupation_controller.py","lineNumber":150,"sourceCode":"                module = modules[name]\n                src_device_map[name] = _get_module_device(module)\n                if device.startswith(\"cpu\"):\n                    _module_to_pinned_cpu(module)\n                else:\n                    module.to(device, non_blocking=True)\n                moved.append(name)\n                _move_unregistered_tensors(module, device)\n            torch.cuda.synchronize()\n        except Exception as e:\n            logger.warning(\n                f\"[_move_modules] move failed, rollback started: target={device} moved={moved} error={e}\",\n            )\n            for name in moved:\n                module = modules.get(name)\n                src_dev = src_device_map.get(name)\n                module.to(src_dev)\n                _move_unregistered_tensors(module, src_dev)\n            raise RuntimeError(\n                f\"failed to move modules to {device}; rollback finished: error={e}\"\n            ) from e\n\n    def _offload_active_modules_to_cpu(self) -> dict[str, str]:\n        restore_map: dict[str, str] = {}\n        for name, module in get_updatable_modules(self.pipeline).items():\n            if _is_layerwise_offload_managed(module):\n                continue\n            device = _get_module_device(module)\n            if not device.startswith(\"cpu\"):\n                restore_map[name] = device\n\n        self._move_modules(list(restore_map.keys()), \"cpu\")\n        self._clear_torch_device_cache()\n        return restore_map\n\n    def _restore_modules_to_original_devices(\n        self, module_device_map: dict[str, str]","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/managers/memory_managers/memory_occupation_controller.py#L132-L168","documentation":"RuntimeError raised by MemoryOccupationController._move_modules after it caught an exception while moving modules to a target device; it rolls back all already-moved modules to their source devices and then re-raises with the original error chained. The message means the move failed midway and state was restored.","triggerScenarios":"Calling offload (e.g. release_memory_occupation/_offload_active_modules_to_cpu) or restore while a module.to(device) throws — commonly CUDA OOM when restoring to GPU, invalid device id, or a tensor on an incompatible device/dtype.","commonSituations":"GPU out of memory when waking the model back up; wrong CUDA_VISIBLE_DEVICES mapping between offload and restore; mixed-precision/dt device ('dt' meta device) modules that can't be .to()'d to cuda; version changes in how dtensors are registered.","solutions":["Free GPU memory (delete other processes/caches, torch.cuda.empty_cache()) and retry the restore","Verify the target device string is valid and visible (cuda:0 within CUDA_VISIBLE_DEVICES)","Check the chained 'error={e}' for the root cause and fix that (OOM, meta-device tensors, etc.)","If using dtensor/FSDP modules, ensure _move_unregistered_tensors handles them or disable that path"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"free, _ = torch.cuda.mem_get_info()\nrequired = estimate_model_bytes()\nassert free > required, f\"need ~{required/1e9:.1f}GB, only {free/1e9:.1f}GB free\"","typeGuard":null,"tryCatchPattern":"try:\n    controller.resume_memory_occupation()\nexcept RuntimeError as e:\n    if \"rollback finished\" in str(e):\n        logger.error(\"move failed, modules rolled back: %s\", e.__cause__)\n        torch.cuda.empty_cache()\n        raise","preventionTips":["Check free VRAM before wake/restore","Pin CUDA_VISIBLE_DEVICES consistently across offload/restore","Inspect the chained cause exception for the real failure"],"tags":["cuda-oom","device-movement","rollback","memory-offload"],"backgroundTag":"cuda-out-of-memory","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}