{"record":{"id":"f27a721b4315f6a0","repo":"sgl-project/sglang","slug":"weight-cache-daemon-for-pp-rank-pp-rank-tp-rank","errorCode":null,"errorMessage":"Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank} did not become ready within {timeout}s","messagePattern":"Weight cache daemon for pp_rank=(.+?) tp_rank=(.+?) did not become ready within (.+?)s","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/entrypoints/engine.py","lineNumber":763,"sourceCode":"\n                daemon_procs.append(proc)\n\n        # Wait for all daemons to be ready (ready file exists). On any failure\n        # (readiness timeout or a daemon exiting early) terminate the siblings\n        # we already spawned before propagating, so a partial launch does not\n        # leak GPU-resident daemons.\n        timeout = server_args.weight_cache_timeout\n        check_interval = 2\n        start_time = time.time()\n        try:\n            for pp_rank in pp_rank_range:\n                for tp_rank in tp_rank_range:\n                    global_rank = compute_global_rank(tp_size, pp_rank, tp_rank)\n                    ready_path = get_ready_path(global_rank)\n                    while not os.path.exists(ready_path):\n                        time.sleep(check_interval)\n                        if time.time() - start_time > timeout:\n                            raise TimeoutError(\n                                f\"Weight cache daemon for pp_rank={pp_rank} \"\n                                f\"tp_rank={tp_rank} did not become ready \"\n                                f\"within {timeout}s\"\n                            )\n                        # Check if daemon process is still alive\n                        for p in daemon_procs:\n                            if not p.is_alive():\n                                raise RuntimeError(\n                                    f\"Weight cache daemon (pid={p.pid}) exited prematurely \"\n                                    f\"with code {p.exitcode}\"\n                                )\n                    logger.info(\n                        f\"Weight cache daemon for pp_rank={pp_rank} \"\n                        f\"tp_rank={tp_rank} is ready\"\n                    )\n        except BaseException:\n            cls._terminate_weight_cache_daemons(daemon_procs)\n            raise","sourceCodeStart":745,"sourceCodeEnd":781,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/entrypoints/engine.py#L745-L781","documentation":"After spawning weight-cache daemon processes, the engine polls for each rank's readiness file. If the file for a given (pp_rank, tp_rank) doesn't appear within the timeout, a TimeoutError is raised — the daemon started but never signaled readiness (slow model download, cold cache, hung init).","triggerScenarios":"_launch_subprocesses with weight-cache daemons enabled; a daemon for some pp_rank/tp_rank fails to write its ready file within the configured timeout, e.g. first-run multi-GB weight download, slow NFS/disk, or network throttling.","commonSituations":"First launch on a new node downloading weights into the cache; shared filesystem latency; undersized default timeout on large models; overloaded nodes during concurrent multi-instance starts.","solutions":["Retry the launch — the partially populated cache makes the next attempt faster (transient cold-start case).","Increase the readiness timeout via the corresponding server_args/env knob if available, or warm the weight cache beforehand (e.g. run a download/prefill step).","Check daemon logs for the underlying slow/hung step (download stall, disk full, permission errors on the cache dir).","Fix storage: point the cache at faster local disk or pre-sync the shared cache across nodes."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import os\nfrom sglang.srt.utils import get_ready_path  # if exposed\n# pre-warm: ensure weights are downloadable/cached before engine start\n# and confirm cache dir is writable and on fast disk\nassert os.access(cache_dir, os.W_OK)","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        engine = sgl.Engine(**kwargs)\n        break\n    except TimeoutError as e:\n        if \"weight cache daemon\" not in str(e) or attempt == 2:\n            raise\n        time.sleep(30)  # cache warms between attempts","preventionTips":["Pre-download/populate the weight cache before multi-instance launch.","Put the weight cache on local NVMe, not slow network storage.","Monitor disk and download bandwidth during first launch; raise timeout if needed."],"tags":["sglang","weight-cache","timeout","startup","daemon"],"backgroundTag":"service-startup-timeout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}