{"record":{"id":"71050699be59f249","repo":"sgl-project/sglang","slug":"waiting-for-main-node-timeout","errorCode":null,"errorMessage":"Waiting for main node timeout!","messagePattern":"Waiting for main node timeout!","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"python/sglang/compile_deep_gemm.py","lineNumber":172,"sourceCode":"                            else [list(base_ids) for _ in range(dp_size)]\n                        )\n\n                    response = requests.post(\n                        f\"{base_url}/generate\",\n                        json=payload,\n                        timeout=600,\n                    )\n                    if response.status_code != 200:\n                        error = response.json()\n                        raise RuntimeError(f\"Sync request failed: {error}\")\n                # Other nodes should wait for the exit signal from Rank-0 node.\n                else:\n                    start_time_waiting = time.perf_counter()\n                    while proc.is_alive():\n                        if time.perf_counter() - start_time_waiting < timeout:\n                            time.sleep(10)\n                        else:\n                            raise TimeoutError(\"Waiting for main node timeout!\")\n                return proc\n        except requests.RequestException:\n            pass\n        time.sleep(10)\n    raise TimeoutError(\n        \"DeepGEMM Kernels compilation timeout.\"\n        \"\\n\\nFeel free and please restart the command.\"\n    )\n\n\ndef compile_server_args(args, compile_args: CompileArgs) -> ServerArgs:\n    \"\"\"The config this script serves with: no cuda graph, no torch compile, and a\n    watchdog that outlives the compilation.\"\"\"\n    args.enable_torch_compile = False\n    # The convenience flags lose to an explicit --cuda-graph-config JSON, which\n    # resolution applies last, so this tool's \"no cuda graph\" guarantee is\n    # merged into that JSON instead -- an operator serving with their own config\n    # still compiles without capture.","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/compile_deep_gemm.py#L154-L190","documentation":"Raised in the DeepGEMM compilation launcher on non-rank-0 nodes: they poll the rank-0 server process, and if it is still alive after `timeout` seconds elapse, the wait loop aborts with TimeoutError('Waiting for main node timeout!'). It indicates rank 0 is compiling (or stuck) longer than the configured wait budget.","triggerScenarios":"Multi-node run_compile where rank 0's DeepGEMM compilation exceeds the timeout value; a hung rank-0 process (deadlocked compile, NFS-slow cache dir); slow first-time compilation of a large kernel set.","commonSituations":"First compilation on cold cache across many nodes; shared filesystem contention on the DeepGEMM cache; too-small timeout passed via compile args.","solutions":["Increase the timeout argument to the compile routine / env var controlling it and re-run (compilation is resumable via cache).","Pre-warm the DeepGEMM cache on rank 0 first, then run the multi-node compile.","Check rank-0 logs to confirm it is progressing, not hung; if hung, clear the cache and restart."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if time_budget < expected_cold_compile_seconds:\n    time_budget *= 2  # raise timeout before entering the wait loop","typeGuard":null,"tryCatchPattern":"try:\n    run_compile(timeout=t)\nexcept TimeoutError as e:\n    if \"main node\" in str(e):\n        run_compile(timeout=t * 2)  # cache makes retry cheaper","preventionTips":["Set generous timeouts for first-ever compilation runs.","Pre-warm rank 0's DeepGEMM cache to shorten others' wait.","Monitor rank-0 logs to distinguish slow progress from a hang."],"tags":["sglang","deep-gemm","timeout","multi-node","compilation"],"backgroundTag":"compilation-timeout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}