sgl-project/sglang · error · RuntimeError

Server is sleeping. Call resume_memory_occupation first.

Error message

Server is sleeping. Call resume_memory_occupation first.

What it means

Raised by Scheduler._handle_generation when generation requests are dispatched while worker.is_sleeping() is true. The model weights are offloaded, so no forward pass can run until the server is woken.

Source

Thrown at python/sglang/multimodal_gen/runtime/managers/scheduler.py:281

    def _dispatch_items(
        self, items: list[tuple[bytes | None, Any]]
    ) -> OutputBatch | list[OutputBatch] | _SequentiallyReturnedOutputs:
        """Dispatch ready queue items; several plain `Req`s form one dynamic batch."""
        reqs = [item[1] for item in items]
        if len(reqs) > 1 and all(isinstance(req, Req) for req in reqs):
            return self._handle_generation(reqs, allow_dynamic_batching=True)
        if len(reqs) > 1:
            return [self._dispatch_single_request(req) for req in reqs]
        return self._dispatch_single_request(reqs[0])

    def _handle_generation(
        self, reqs: list[Any], *, allow_dynamic_batching: bool = True
    ):
        """Dispatch generation requests, merging compatible requests when allowed."""
        reqs = self._normalize_generation_reqs(reqs)
        if self.worker.is_sleeping():
            raise RuntimeError(
                "Server is sleeping. Call resume_memory_occupation first."
            )
        warmup_reqs = [req for req in reqs if req.is_warmup]
        if warmup_reqs:
            self._ensure_warmup_progress_bar(warmup_reqs[0])

        # Use the head request trace context for scheduler-side dispatch work.
        req = reqs[0]
        req.trace_ctx.rebuild_thread_context()
        with trace_slice(
            req.trace_ctx,
            DiffStage.SCHEDULER_DISPATCH,
            thread_finish_flag=True,
        ):
            if (
                len(reqs) == 1
                and self.server_args.pipeline_config.supports_sequential_multi_output_inference()
                and max(1, int(req.num_outputs_per_prompt or 1)) > 1

View on GitHub (pinned to 0132848349)

Solutions

1. Call resume_memory_occupation before sending generation requests
2. Buffer/queue client requests while sleeping and flush after wake
3. Make clients retry on this RuntimeError after triggering wake

Example fix

// before
controller.release_memory_occupation()
server.generate(req)  # raises
// after
controller.resume_memory_occupation()
server.generate(req)

assert not worker.is_sleeping(), "server asleep; resume_memory_occupation first"

try:
    server.generate(req)
except RuntimeError as e:
    if "Server is sleeping" in str(e):
        controller.resume_memory_occupation(); retry(req)
    else:
        raise

When it happens

Trigger: Sending generate/warmup requests to a server that has executed release_memory_occupation (sleep) without a subsequent resume_memory_occupation.

Common situations: Sleeping the server to free VRAM for training and then letting clients/inference requests hit it before wake; warmup requests queued during sleep; health-check traffic arriving mid-sleep.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• sleep/wake does not support FSDP inference
• Cannot update weights while the server is sleeping. Call res

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1296f67b5f3a2902. Report an issue: GitHub.