sgl-project/sglang · critical · RuntimeError

Scheduler terminated after {self._max_consecutive_errors} co

Error message

Scheduler terminated after {self._max_consecutive_errors} consecutive errors. Last error: {e}

What it means

RuntimeError raised at the top of Scheduler.event_loop after self._max_consecutive_errors consecutive iteration failures; the last underlying exception is chained as the cause. It deliberately kills the scheduler process instead of spinning forever.

Source

Thrown at python/sglang/multimodal_gen/runtime/managers/scheduler.py:1214

                now = time.monotonic()
                self.waiting_queue.extend(
                    [(identity, req, now) for identity, req in new_reqs]
                )
                # Reset error count on success
                self._consecutive_error_count = 0
            except Exception as e:
                self._consecutive_error_count += 1
                logger.error(
                    f"Error receiving requests in scheduler event loop "
                    f"(attempt {self._consecutive_error_count}/{self._max_consecutive_errors}): {e}",
                    exc_info=True,
                )
                if self._consecutive_error_count >= self._max_consecutive_errors:
                    logger.error(
                        f"Maximum consecutive errors ({self._max_consecutive_errors}) reached. "
                        "Terminating scheduler event loop."
                    )
                    raise RuntimeError(
                        f"Scheduler terminated after {self._max_consecutive_errors} "
                        f"consecutive errors. Last error: {e}"
                    ) from e
                continue

            # 2: execute, make sure a reply is always sent
            items = self.get_next_batch_to_run()
            if not items:
                if self.waiting_queue and self._dynamic_batching_enabled():
                    oldest_ts = self.waiting_queue[0][2]
                    elapsed_ms = (time.monotonic() - oldest_ts) * 1000.0
                    remaining_ms = max(0, self._batching_delay_s * 1000.0 - elapsed_ms)
                    if remaining_ms > 0 and self.receiver is not None:
                        self._poller.poll(timeout=remaining_ms)
                    elif remaining_ms > 0:
                        time.sleep(remaining_ms / 1000.0)
                continue

View on GitHub (pinned to 0132848349)

Solutions

Inspect the chained 'Last error' and the logger.error line above it for the root cause and fix that exception
Check scheduler logs for the repeated traceback pattern to identify the failing stage (recv vs execute)
If failures are transient/environmental, raise _max_consecutive_errors or add backoff/retry
Restart the scheduler process after fixing the underlying issue

Defensive patterns

Strategy: retry

Try / catch

try:
    run_scheduler_process(args)
except RuntimeError as e:
    if "consecutive errors" in str(e):
        log_chained_cause(e); restart_with_backoff()  # after fixing root cause

Prevention

Monitor consecutive_error_count metric and alert before the cap
Keep the chained cause (raise ... from e) when re-raising
Fix the first repeated traceback, not the terminator

When it happens

Trigger: Any recurring exception inside the event loop (bad request handling, IPC errors, CUDA errors) repeated _max_consecutive_errors times in a row triggers termination; run_scheduler_process then exits.

Common situations: A malformed request type crashing every iteration; CUDA device in a bad state; broken IPC channel to workers; each iteration failing identically after a config mistake.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eb89663fdd712443. Report an issue: GitHub.