sgl-project/sglang · critical · TimeoutError

DeepGEMM Kernels compilation timeout.\n\nFeel free and…

Error message

DeepGEMM Kernels compilation timeout.\n\nFeel free and please restart the command.

What it means

Terminal failure of the retry loop in launch_server_process_and_send_one_request: after repeatedly launching the server and attempting the /generate request, requests.RequestException kept occurring (server never became reachable/ready) until the overall deadline passed. The message explicitly invites restarting, since DeepGEMM compilation may have made partial progress that a retry can reuse.

Solutions

Simply re-run the command as the message suggests — the DeepGEMM cache preserves progress and a later attempt often succeeds.
Check that the port used by the compile server is free and the model path exists.
Look at the captured server process output for the startup crash reason and fix it (args, CUDA device visibility).

Defensive patterns

Strategy: retry

Validate before calling

import socket, subprocess
subprocess.run(["fuser", "-k", f"{port}/tcp"], check=False)  # free the port before compiling

Try / catch

for attempt in range(2):
    try:
        run_compile(...); break
    except TimeoutError as e:
        if "restart" not in str(e): raise
        time.sleep(30)  # partial kernel cache persists; retry is cheap

Prevention

Check the compile-server port is free and the model path exists before launching.
Keep the DeepGEMM cache dir on fast local storage.
Run the single-node compile once successfully before multi-node attempts.

When it happens

Trigger: run_compile where the server process never comes up (port conflicts, crash on startup, missing weights) so every HTTP attempt raises RequestException; loop exhausts its retry budget.

Common situations: Server port already in use; model path missing; startup crash due to bad args; firewall/port issues between the launcher and the server.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/66d7d588d20b390e. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/compile_deep_gemm.py:177

                        json=payload,
                        timeout=600,
                    )
                    if response.status_code != 200:
                        error = response.json()
                        raise RuntimeError(f"Sync request failed: {error}")
                # Other nodes should wait for the exit signal from Rank-0 node.
                else:
                    start_time_waiting = time.perf_counter()
                    while proc.is_alive():
                        if time.perf_counter() - start_time_waiting < timeout:
                            time.sleep(10)
                        else:
                            raise TimeoutError("Waiting for main node timeout!")
                return proc
        except requests.RequestException:
            pass
        time.sleep(10)
    raise TimeoutError(
        "DeepGEMM Kernels compilation timeout."
        "\n\nFeel free and please restart the command."
    )


def compile_server_args(args, compile_args: CompileArgs) -> ServerArgs:
    """The config this script serves with: no cuda graph, no torch compile, and a
    watchdog that outlives the compilation."""
    args.enable_torch_compile = False
    # The convenience flags lose to an explicit --cuda-graph-config JSON, which
    # resolution applies last, so this tool's "no cuda graph" guarantee is
    # merged into that JSON instead -- an operator serving with their own config
    # still compiles without capture.
    explicit = args.cuda_graph_config
    if isinstance(explicit, CudaGraphConfig):
        explicit = explicit.to_dict()
    explicit = dict(explicit or {})
    for phase in (Phase.DECODE, Phase.PREFILL):

View on GitHub (pinned to 0132848349)