sgl-project/sglang · critical · RuntimeError

Weight cache daemon (pid={p.pid}) exited prematurely with co

Error message

Weight cache daemon (pid={p.pid}) exited prematurely with code {p.exitcode}

What it means

While waiting for weight-cache daemon readiness, the engine periodically checks that every spawned daemon process is still alive. If any daemon exits before writing its ready file, a RuntimeError is raised with its pid and exit code — the daemon crashed rather than merely being slow.

Source

Thrown at python/sglang/srt/entrypoints/engine.py:771

        check_interval = 2
        start_time = time.time()
        try:
            for pp_rank in pp_rank_range:
                for tp_rank in tp_rank_range:
                    global_rank = compute_global_rank(tp_size, pp_rank, tp_rank)
                    ready_path = get_ready_path(global_rank)
                    while not os.path.exists(ready_path):
                        time.sleep(check_interval)
                        if time.time() - start_time > timeout:
                            raise TimeoutError(
                                f"Weight cache daemon for pp_rank={pp_rank} "
                                f"tp_rank={tp_rank} did not become ready "
                                f"within {timeout}s"
                            )
                        # Check if daemon process is still alive
                        for p in daemon_procs:
                            if not p.is_alive():
                                raise RuntimeError(
                                    f"Weight cache daemon (pid={p.pid}) exited prematurely "
                                    f"with code {p.exitcode}"
                                )
                    logger.info(
                        f"Weight cache daemon for pp_rank={pp_rank} "
                        f"tp_rank={tp_rank} is ready"
                    )
        except BaseException:
            cls._terminate_weight_cache_daemons(daemon_procs)
            raise

        logger.info(
            f"All {num_daemons} weight cache daemons on node "
            f"{server_args.node_rank} are ready"
        )
        return daemon_procs

    @staticmethod

View on GitHub (pinned to 0132848349)

Solutions

Inspect the exit code: negative codes mean killed by signal (e.g. -9 OOM → free memory or lower memory usage); 1 usually means a Python exception — read the daemon's log/stderr.
Fix root causes: writable cache directory, matching CUDA/driver versions, adequate free memory.
Clear stale ready files / cache locks from prior crashed runs before relaunching.
Re-run the launch after remediation; if it recurs, launch one daemon manually with the same args to see the traceback.

Defensive patterns

Strategy: try-catch

Validate before calling

# before launch: verify cache dir writable and no stale ready files
import os, glob
for f in glob.glob(f"{cache_dir}/**/ready*", recursive=True):
    os.remove(f)  # clear stale readiness markers
assert os.access(os.path.dirname(cache_dir), os.W_OK)

Try / catch

try:
    engine = sgl.Engine(**kwargs)
except RuntimeError as e:
    if "exited prematurely" in str(e):
        # parse pid/exitcode, surface daemon logs, check dmesg for OOM
        raise RuntimeError(f"daemon crash: {e}; check dmesg -T | grep -i kill")
    raise

Prevention

Provision enough free memory so OOM killer can't reap daemons during weight load.
Keep sglang/CUDA/driver versions consistent across the host.
Clean stale cache/ready files between crashed runs.

When it happens

Trigger: A weight-cache daemon subprocess dies during startup (import error, OOM kill, bad cache dir permissions, port/file conflicts) before its ready file appears, inside _launch_weight_cache_daemons.

Common situations: OOM killer reaping daemons on memory-tight nodes; CUDA/driver mismatch causing import failure in the subprocess; unwritable or colliding cache paths; leftover stale ready files from a previous crashed run confusing state.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/41cda5f7c7caf4ef. Report an issue: GitHub.