sgl-project/sglang · error · RuntimeError

Cannot update weights while the server is sleeping. Call res

Error message

Cannot update weights while the server is sleeping. Call resume_memory_occupation first.

What it means

Raised by the scheduler's _handle_update_weights_from_disk when an update_weights_from_disk request arrives while worker.is_sleeping() is true (weights offloaded to CPU). Weight files are not resident on GPU, so a disk-based weight update cannot proceed.

Source

Thrown at python/sglang/multimodal_gen/runtime/managers/scheduler.py:239

    def _handle_unmerge_lora(self, reqs: List[Any]) -> OutputBatch:
        req = reqs[0]
        return self.worker.unmerge_lora_weights(req.target)

    def _handle_list_loras(self, _reqs: List[Any]) -> OutputBatch:
        return self.worker.list_loras()

    def _handle_shutdown(self, _reqs: List[Any]) -> OutputBatch:
        self._running = False
        return OutputBatch()

    def _handle_release_realtime_session(self, reqs: List[Any]) -> OutputBatch:
        req = reqs[0]
        return self.worker.release_realtime_session(req.session_id)

    def _handle_update_weights_from_disk(self, reqs: List[Any]) -> OutputBatch:
        """Handle update_weights_from_disk request for RL workflows."""
        if self.worker.is_sleeping():
            raise RuntimeError(
                "Cannot update weights while the server is sleeping. "
                "Call resume_memory_occupation first."
            )
        return super()._handle_update_weights_from_disk(reqs)

    @staticmethod
    def _normalize_generation_reqs(reqs: list[Any]) -> list[Req]:
        if len(reqs) == 1 and isinstance(reqs[0], list):
            return reqs[0]
        return reqs

    def _dispatch_single_request(self, req_or_group: Any) -> OutputBatch:
        if isinstance(req_or_group, list):
            if not all(isinstance(req, Req) for req in req_or_group):
                return OutputBatch(
                    error=f"Unknown request group type: {type(req_or_group)}"
                )
            return self._handle_generation(req_or_group, allow_dynamic_batching=False)

View on GitHub (pinned to 0132848349)

Solutions

Call resume_memory_occupation (wake) before issuing update_weights_from_disk
Reorder the RL loop: wake -> update weights -> optionally sleep again

Example fix

// before
controller.release_memory_occupation()
client.update_weights_from_disk(...)  # raises
// after
controller.resume_memory_occupation()
client.update_weights_from_disk(...)

Defensive patterns

Strategy: validation

Validate before calling

if worker.is_sleeping():
    controller.resume_memory_occupation()
client.update_weights_from_disk(path)

Try / catch

try:
    client.update_weights_from_disk(path)
except RuntimeError as e:
    if "sleeping" in str(e):
        controller.resume_memory_occupation()
        client.update_weights_from_disk(path)
    else:
        raise

Prevention

Order RL loop as wake -> update -> sleep
Assert not is_sleeping() before weight updates

When it happens

Trigger: RL workflow calls update_weights_from_disk after release_memory_occupation/sleep and before resume_memory_occupation/wake.

Common situations: RL training loop that sleeps the server between rollouts to free VRAM and forgets to wake it before pushing new checkpoints; race where the sleep request is processed before the update request.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/16c271f242d834e3. Report an issue: GitHub.