sgl-project/sglang · error · TimeoutError

Waiting for main node timeout!

Error message

Waiting for main node timeout!

What it means

Raised in the DeepGEMM compilation launcher on non-rank-0 nodes: they poll the rank-0 server process, and if it is still alive after `timeout` seconds elapse, the wait loop aborts with TimeoutError('Waiting for main node timeout!'). It indicates rank 0 is compiling (or stuck) longer than the configured wait budget.

Source

Thrown at python/sglang/compile_deep_gemm.py:172

                            else [list(base_ids) for _ in range(dp_size)]
                        )

                    response = requests.post(
                        f"{base_url}/generate",
                        json=payload,
                        timeout=600,
                    )
                    if response.status_code != 200:
                        error = response.json()
                        raise RuntimeError(f"Sync request failed: {error}")
                # Other nodes should wait for the exit signal from Rank-0 node.
                else:
                    start_time_waiting = time.perf_counter()
                    while proc.is_alive():
                        if time.perf_counter() - start_time_waiting < timeout:
                            time.sleep(10)
                        else:
                            raise TimeoutError("Waiting for main node timeout!")
                return proc
        except requests.RequestException:
            pass
        time.sleep(10)
    raise TimeoutError(
        "DeepGEMM Kernels compilation timeout."
        "\n\nFeel free and please restart the command."
    )


def compile_server_args(args, compile_args: CompileArgs) -> ServerArgs:
    """The config this script serves with: no cuda graph, no torch compile, and a
    watchdog that outlives the compilation."""
    args.enable_torch_compile = False
    # The convenience flags lose to an explicit --cuda-graph-config JSON, which
    # resolution applies last, so this tool's "no cuda graph" guarantee is
    # merged into that JSON instead -- an operator serving with their own config
    # still compiles without capture.

View on GitHub (pinned to 0132848349)

Solutions

1. Increase the timeout argument to the compile routine / env var controlling it and re-run (compilation is resumable via cache).
2. Pre-warm the DeepGEMM cache on rank 0 first, then run the multi-node compile.
3. Check rank-0 logs to confirm it is progressing, not hung; if hung, clear the cache and restart.

if time_budget < expected_cold_compile_seconds:
    time_budget *= 2  # raise timeout before entering the wait loop

try:
    run_compile(timeout=t)
except TimeoutError as e:
    if "main node" in str(e):
        run_compile(timeout=t * 2)  # cache makes retry cheaper

When it happens

Trigger: Multi-node run_compile where rank 0's DeepGEMM compilation exceeds the timeout value; a hung rank-0 process (deadlocked compile, NFS-slow cache dir); slow first-time compilation of a large kernel set.

Common situations: First compilation on cold cache across many nodes; shared filesystem contention on the DeepGEMM cache; too-small timeout passed via compile args.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• DeepGEMM Kernels compilation timeout.\n\nFeel free and pleas
• Sync request failed: {error}
• Server failed to start within the timeout period.
• Multi-node weight cache daemons (nnodes > 1) require --dist-
• Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/71050699be59f249. Report an issue: GitHub.