{"record":{"id":"41cda5f7c7caf4ef","repo":"sgl-project/sglang","slug":"weight-cache-daemon-pid-p-pid-exited-premature","errorCode":null,"errorMessage":"Weight cache daemon (pid={p.pid}) exited prematurely with code {p.exitcode}","messagePattern":"Weight cache daemon \\(pid=(.+?)\\) exited prematurely with code (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/entrypoints/engine.py","lineNumber":771,"sourceCode":"        check_interval = 2\n        start_time = time.time()\n        try:\n            for pp_rank in pp_rank_range:\n                for tp_rank in tp_rank_range:\n                    global_rank = compute_global_rank(tp_size, pp_rank, tp_rank)\n                    ready_path = get_ready_path(global_rank)\n                    while not os.path.exists(ready_path):\n                        time.sleep(check_interval)\n                        if time.time() - start_time > timeout:\n                            raise TimeoutError(\n                                f\"Weight cache daemon for pp_rank={pp_rank} \"\n                                f\"tp_rank={tp_rank} did not become ready \"\n                                f\"within {timeout}s\"\n                            )\n                        # Check if daemon process is still alive\n                        for p in daemon_procs:\n                            if not p.is_alive():\n                                raise RuntimeError(\n                                    f\"Weight cache daemon (pid={p.pid}) exited prematurely \"\n                                    f\"with code {p.exitcode}\"\n                                )\n                    logger.info(\n                        f\"Weight cache daemon for pp_rank={pp_rank} \"\n                        f\"tp_rank={tp_rank} is ready\"\n                    )\n        except BaseException:\n            cls._terminate_weight_cache_daemons(daemon_procs)\n            raise\n\n        logger.info(\n            f\"All {num_daemons} weight cache daemons on node \"\n            f\"{server_args.node_rank} are ready\"\n        )\n        return daemon_procs\n\n    @staticmethod","sourceCodeStart":753,"sourceCodeEnd":789,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/entrypoints/engine.py#L753-L789","documentation":"While waiting for weight-cache daemon readiness, the engine periodically checks that every spawned daemon process is still alive. If any daemon exits before writing its ready file, a RuntimeError is raised with its pid and exit code — the daemon crashed rather than merely being slow.","triggerScenarios":"A weight-cache daemon subprocess dies during startup (import error, OOM kill, bad cache dir permissions, port/file conflicts) before its ready file appears, inside _launch_weight_cache_daemons.","commonSituations":"OOM killer reaping daemons on memory-tight nodes; CUDA/driver mismatch causing import failure in the subprocess; unwritable or colliding cache paths; leftover stale ready files from a previous crashed run confusing state.","solutions":["Inspect the exit code: negative codes mean killed by signal (e.g. -9 OOM → free memory or lower memory usage); 1 usually means a Python exception — read the daemon's log/stderr.","Fix root causes: writable cache directory, matching CUDA/driver versions, adequate free memory.","Clear stale ready files / cache locks from prior crashed runs before relaunching.","Re-run the launch after remediation; if it recurs, launch one daemon manually with the same args to see the traceback."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# before launch: verify cache dir writable and no stale ready files\nimport os, glob\nfor f in glob.glob(f\"{cache_dir}/**/ready*\", recursive=True):\n    os.remove(f)  # clear stale readiness markers\nassert os.access(os.path.dirname(cache_dir), os.W_OK)","typeGuard":null,"tryCatchPattern":"try:\n    engine = sgl.Engine(**kwargs)\nexcept RuntimeError as e:\n    if \"exited prematurely\" in str(e):\n        # parse pid/exitcode, surface daemon logs, check dmesg for OOM\n        raise RuntimeError(f\"daemon crash: {e}; check dmesg -T | grep -i kill\")\n    raise","preventionTips":["Provision enough free memory so OOM killer can't reap daemons during weight load.","Keep sglang/CUDA/driver versions consistent across the host.","Clean stale cache/ready files between crashed runs."],"tags":["sglang","weight-cache","daemon-crash","startup"],"backgroundTag":"subprocess-exited-unexpectedly","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}