{"record":{"id":"16c271f242d834e3","repo":"sgl-project/sglang","slug":"cannot-update-weights-while-the-server-is-sleeping","errorCode":null,"errorMessage":"Cannot update weights while the server is sleeping. Call resume_memory_occupation first.","messagePattern":"Cannot update weights while the server is sleeping\\. Call resume_memory_occupation first\\.","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/managers/scheduler.py","lineNumber":239,"sourceCode":"    def _handle_unmerge_lora(self, reqs: List[Any]) -> OutputBatch:\n        req = reqs[0]\n        return self.worker.unmerge_lora_weights(req.target)\n\n    def _handle_list_loras(self, _reqs: List[Any]) -> OutputBatch:\n        return self.worker.list_loras()\n\n    def _handle_shutdown(self, _reqs: List[Any]) -> OutputBatch:\n        self._running = False\n        return OutputBatch()\n\n    def _handle_release_realtime_session(self, reqs: List[Any]) -> OutputBatch:\n        req = reqs[0]\n        return self.worker.release_realtime_session(req.session_id)\n\n    def _handle_update_weights_from_disk(self, reqs: List[Any]) -> OutputBatch:\n        \"\"\"Handle update_weights_from_disk request for RL workflows.\"\"\"\n        if self.worker.is_sleeping():\n            raise RuntimeError(\n                \"Cannot update weights while the server is sleeping. \"\n                \"Call resume_memory_occupation first.\"\n            )\n        return super()._handle_update_weights_from_disk(reqs)\n\n    @staticmethod\n    def _normalize_generation_reqs(reqs: list[Any]) -> list[Req]:\n        if len(reqs) == 1 and isinstance(reqs[0], list):\n            return reqs[0]\n        return reqs\n\n    def _dispatch_single_request(self, req_or_group: Any) -> OutputBatch:\n        if isinstance(req_or_group, list):\n            if not all(isinstance(req, Req) for req in req_or_group):\n                return OutputBatch(\n                    error=f\"Unknown request group type: {type(req_or_group)}\"\n                )\n            return self._handle_generation(req_or_group, allow_dynamic_batching=False)","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/managers/scheduler.py#L221-L257","documentation":"Raised by the scheduler's _handle_update_weights_from_disk when an update_weights_from_disk request arrives while worker.is_sleeping() is true (weights offloaded to CPU). Weight files are not resident on GPU, so a disk-based weight update cannot proceed.","triggerScenarios":"RL workflow calls update_weights_from_disk after release_memory_occupation/sleep and before resume_memory_occupation/wake.","commonSituations":"RL training loop that sleeps the server between rollouts to free VRAM and forgets to wake it before pushing new checkpoints; race where the sleep request is processed before the update request.","solutions":["Call resume_memory_occupation (wake) before issuing update_weights_from_disk","Reorder the RL loop: wake -> update weights -> optionally sleep again"],"exampleFix":"// before\ncontroller.release_memory_occupation()\nclient.update_weights_from_disk(...)  # raises\n// after\ncontroller.resume_memory_occupation()\nclient.update_weights_from_disk(...)","handlingStrategy":"validation","validationCode":"if worker.is_sleeping():\n    controller.resume_memory_occupation()\nclient.update_weights_from_disk(path)","typeGuard":null,"tryCatchPattern":"try:\n    client.update_weights_from_disk(path)\nexcept RuntimeError as e:\n    if \"sleeping\" in str(e):\n        controller.resume_memory_occupation()\n        client.update_weights_from_disk(path)\n    else:\n        raise","preventionTips":["Order RL loop as wake -> update -> sleep","Assert not is_sleeping() before weight updates"],"tags":["sleep-wake","weight-update","rl-workflow"],"backgroundTag":"invalid-state-transition","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}