sgl-project/sglang · error · TimeoutError

Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank}

Error message

Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank} did not become ready within {timeout}s

What it means

After spawning weight-cache daemon processes, the engine polls for each rank's readiness file. If the file for a given (pp_rank, tp_rank) doesn't appear within the timeout, a TimeoutError is raised — the daemon started but never signaled readiness (slow model download, cold cache, hung init).

Source

Thrown at python/sglang/srt/entrypoints/engine.py:763

                daemon_procs.append(proc)

        # Wait for all daemons to be ready (ready file exists). On any failure
        # (readiness timeout or a daemon exiting early) terminate the siblings
        # we already spawned before propagating, so a partial launch does not
        # leak GPU-resident daemons.
        timeout = server_args.weight_cache_timeout
        check_interval = 2
        start_time = time.time()
        try:
            for pp_rank in pp_rank_range:
                for tp_rank in tp_rank_range:
                    global_rank = compute_global_rank(tp_size, pp_rank, tp_rank)
                    ready_path = get_ready_path(global_rank)
                    while not os.path.exists(ready_path):
                        time.sleep(check_interval)
                        if time.time() - start_time > timeout:
                            raise TimeoutError(
                                f"Weight cache daemon for pp_rank={pp_rank} "
                                f"tp_rank={tp_rank} did not become ready "
                                f"within {timeout}s"
                            )
                        # Check if daemon process is still alive
                        for p in daemon_procs:
                            if not p.is_alive():
                                raise RuntimeError(
                                    f"Weight cache daemon (pid={p.pid}) exited prematurely "
                                    f"with code {p.exitcode}"
                                )
                    logger.info(
                        f"Weight cache daemon for pp_rank={pp_rank} "
                        f"tp_rank={tp_rank} is ready"
                    )
        except BaseException:
            cls._terminate_weight_cache_daemons(daemon_procs)
            raise

View on GitHub (pinned to 0132848349)

Solutions

Retry the launch — the partially populated cache makes the next attempt faster (transient cold-start case).
Increase the readiness timeout via the corresponding server_args/env knob if available, or warm the weight cache beforehand (e.g. run a download/prefill step).
Check daemon logs for the underlying slow/hung step (download stall, disk full, permission errors on the cache dir).
Fix storage: point the cache at faster local disk or pre-sync the shared cache across nodes.

Defensive patterns

Strategy: retry

Validate before calling

import os
from sglang.srt.utils import get_ready_path  # if exposed
# pre-warm: ensure weights are downloadable/cached before engine start
# and confirm cache dir is writable and on fast disk
assert os.access(cache_dir, os.W_OK)

Try / catch

for attempt in range(3):
    try:
        engine = sgl.Engine(**kwargs)
        break
    except TimeoutError as e:
        if "weight cache daemon" not in str(e) or attempt == 2:
            raise
        time.sleep(30)  # cache warms between attempts

Prevention

Pre-download/populate the weight cache before multi-instance launch.
Put the weight cache on local NVMe, not slow network storage.
Monitor disk and download bandwidth during first launch; raise timeout if needed.

When it happens

Trigger: _launch_subprocesses with weight-cache daemons enabled; a daemon for some pp_rank/tp_rank fails to write its ready file within the configured timeout, e.g. first-run multi-GB weight download, slow NFS/disk, or network throttling.

Common situations: First launch on a new node downloading weights into the cache; shared filesystem latency; undersized default timeout on large models; overloaded nodes during concurrent multi-instance starts.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f27a721b4315f6a0. Report an issue: GitHub.