{"record":{"id":"eb89663fdd712443","repo":"sgl-project/sglang","slug":"scheduler-terminated-after-self-max-consecutive","errorCode":null,"errorMessage":"Scheduler terminated after {self._max_consecutive_errors} consecutive errors. Last error: {e}","messagePattern":"Scheduler terminated after (.+?) consecutive errors\\. Last error: (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/managers/scheduler.py","lineNumber":1214,"sourceCode":"                now = time.monotonic()\n                self.waiting_queue.extend(\n                    [(identity, req, now) for identity, req in new_reqs]\n                )\n                # Reset error count on success\n                self._consecutive_error_count = 0\n            except Exception as e:\n                self._consecutive_error_count += 1\n                logger.error(\n                    f\"Error receiving requests in scheduler event loop \"\n                    f\"(attempt {self._consecutive_error_count}/{self._max_consecutive_errors}): {e}\",\n                    exc_info=True,\n                )\n                if self._consecutive_error_count >= self._max_consecutive_errors:\n                    logger.error(\n                        f\"Maximum consecutive errors ({self._max_consecutive_errors}) reached. \"\n                        \"Terminating scheduler event loop.\"\n                    )\n                    raise RuntimeError(\n                        f\"Scheduler terminated after {self._max_consecutive_errors} \"\n                        f\"consecutive errors. Last error: {e}\"\n                    ) from e\n                continue\n\n            # 2: execute, make sure a reply is always sent\n            items = self.get_next_batch_to_run()\n            if not items:\n                if self.waiting_queue and self._dynamic_batching_enabled():\n                    oldest_ts = self.waiting_queue[0][2]\n                    elapsed_ms = (time.monotonic() - oldest_ts) * 1000.0\n                    remaining_ms = max(0, self._batching_delay_s * 1000.0 - elapsed_ms)\n                    if remaining_ms > 0 and self.receiver is not None:\n                        self._poller.poll(timeout=remaining_ms)\n                    elif remaining_ms > 0:\n                        time.sleep(remaining_ms / 1000.0)\n                continue\n","sourceCodeStart":1196,"sourceCodeEnd":1232,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/managers/scheduler.py#L1196-L1232","documentation":"RuntimeError raised at the top of Scheduler.event_loop after self._max_consecutive_errors consecutive iteration failures; the last underlying exception is chained as the cause. It deliberately kills the scheduler process instead of spinning forever.","triggerScenarios":"Any recurring exception inside the event loop (bad request handling, IPC errors, CUDA errors) repeated _max_consecutive_errors times in a row triggers termination; run_scheduler_process then exits.","commonSituations":"A malformed request type crashing every iteration; CUDA device in a bad state; broken IPC channel to workers; each iteration failing identically after a config mistake.","solutions":["Inspect the chained 'Last error' and the logger.error line above it for the root cause and fix that exception","Check scheduler logs for the repeated traceback pattern to identify the failing stage (recv vs execute)","If failures are transient/environmental, raise _max_consecutive_errors or add backoff/retry","Restart the scheduler process after fixing the underlying issue"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    run_scheduler_process(args)\nexcept RuntimeError as e:\n    if \"consecutive errors\" in str(e):\n        log_chained_cause(e); restart_with_backoff()  # after fixing root cause","preventionTips":["Monitor consecutive_error_count metric and alert before the cap","Keep the chained cause (raise ... from e) when re-raising","Fix the first repeated traceback, not the terminator"],"tags":["event-loop","crash-loop","scheduler","circuit-breaker"],"backgroundTag":"consecutive-failure-circuit-breaker","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}