{"record":{"id":"66d7d588d20b390e","repo":"sgl-project/sglang","slug":"deepgemm-kernels-compilation-timeout-n-nfeel-free","errorCode":null,"errorMessage":"DeepGEMM Kernels compilation timeout.\\n\\nFeel free and please restart the command.","messagePattern":"DeepGEMM Kernels compilation timeout\\.\\\\n\\\\nFeel free and please restart the command\\.","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"critical","filePath":"python/sglang/compile_deep_gemm.py","lineNumber":177,"sourceCode":"                        json=payload,\n                        timeout=600,\n                    )\n                    if response.status_code != 200:\n                        error = response.json()\n                        raise RuntimeError(f\"Sync request failed: {error}\")\n                # Other nodes should wait for the exit signal from Rank-0 node.\n                else:\n                    start_time_waiting = time.perf_counter()\n                    while proc.is_alive():\n                        if time.perf_counter() - start_time_waiting < timeout:\n                            time.sleep(10)\n                        else:\n                            raise TimeoutError(\"Waiting for main node timeout!\")\n                return proc\n        except requests.RequestException:\n            pass\n        time.sleep(10)\n    raise TimeoutError(\n        \"DeepGEMM Kernels compilation timeout.\"\n        \"\\n\\nFeel free and please restart the command.\"\n    )\n\n\ndef compile_server_args(args, compile_args: CompileArgs) -> ServerArgs:\n    \"\"\"The config this script serves with: no cuda graph, no torch compile, and a\n    watchdog that outlives the compilation.\"\"\"\n    args.enable_torch_compile = False\n    # The convenience flags lose to an explicit --cuda-graph-config JSON, which\n    # resolution applies last, so this tool's \"no cuda graph\" guarantee is\n    # merged into that JSON instead -- an operator serving with their own config\n    # still compiles without capture.\n    explicit = args.cuda_graph_config\n    if isinstance(explicit, CudaGraphConfig):\n        explicit = explicit.to_dict()\n    explicit = dict(explicit or {})\n    for phase in (Phase.DECODE, Phase.PREFILL):","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/compile_deep_gemm.py#L159-L195","documentation":"Terminal failure of the retry loop in launch_server_process_and_send_one_request: after repeatedly launching the server and attempting the /generate request, requests.RequestException kept occurring (server never became reachable/ready) until the overall deadline passed. The message explicitly invites restarting, since DeepGEMM compilation may have made partial progress that a retry can reuse.","triggerScenarios":"run_compile where the server process never comes up (port conflicts, crash on startup, missing weights) so every HTTP attempt raises RequestException; loop exhausts its retry budget.","commonSituations":"Server port already in use; model path missing; startup crash due to bad args; firewall/port issues between the launcher and the server.","solutions":["Simply re-run the command as the message suggests — the DeepGEMM cache preserves progress and a later attempt often succeeds.","Check that the port used by the compile server is free and the model path exists.","Look at the captured server process output for the startup crash reason and fix it (args, CUDA device visibility)."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import socket, subprocess\nsubprocess.run([\"fuser\", \"-k\", f\"{port}/tcp\"], check=False)  # free the port before compiling","typeGuard":null,"tryCatchPattern":"for attempt in range(2):\n    try:\n        run_compile(...); break\n    except TimeoutError as e:\n        if \"restart\" not in str(e): raise\n        time.sleep(30)  # partial kernel cache persists; retry is cheap","preventionTips":["Check the compile-server port is free and the model path exists before launching.","Keep the DeepGEMM cache dir on fast local storage.","Run the single-node compile once successfully before multi-node attempts."],"tags":["sglang","deep-gemm","timeout","server-startup","compilation"],"backgroundTag":"compilation-timeout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}