{"record":{"id":"8805ea3d1cce2913","repo":"sgl-project/sglang","slug":"multi-node-weight-cache-daemons-nnodes-1-requi","errorCode":null,"errorMessage":"Multi-node weight cache daemons (nnodes > 1) require --dist-init-addr so all nodes rendezvous at the same endpoint.","messagePattern":"Multi-node weight cache daemons \\(nnodes > 1\\) require --dist-init-addr so all nodes rendezvous at the same endpoint\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/entrypoints/engine.py","lineNumber":676,"sourceCode":"        All daemon processes join the same NCCL distributed group so that\n        TP-sharded model loading works correctly. Each daemon holds its\n        rank's weight shard in GPU memory and serves IPC handles.\n\n        Lifecycle: these daemons are *co-terminal* with the engine. They are\n        children of this process (kill_itself_when_parent_died installs\n        PR_SET_PDEATHSIG) and are gracefully reaped in ``shutdown()``. They do\n        NOT persist across engine restarts, so ``--weight-cache-mode daemon``\n        on its own does not deliver a faster restart -- the first start is in\n        fact slower (disk-load into the daemon plus the IPC handshake). The\n        fast-recovery story is the standalone launcher\n        (``python -m sglang.srt.weight_cache.daemon``) plus\n        ``--weight-cache-mode client``, where the daemon outlives the engine.\n        \"\"\"\n        # Multi-node needs an explicit rendezvous address; otherwise each node\n        # picks its own local 127.0.0.1 port (below) and the per-node daemons\n        # can never form the joint process group.\n        if server_args.nnodes > 1 and not server_args.dist_init_addr:\n            raise ValueError(\n                \"Multi-node weight cache daemons (nnodes > 1) require \"\n                \"--dist-init-addr so all nodes rendezvous at the same endpoint.\"\n            )\n\n        tp_size = server_args.tp_size\n\n        pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = (\n            _calculate_rank_ranges(\n                server_args.nnodes,\n                get_parallel().pp_size,\n                tp_size,\n                server_args.node_rank,\n            )\n        )\n\n        # Build the distributed init method (multi-node uses the user-provided\n        # dist_init_addr so all nodes reach the same endpoint).\n        if server_args.dist_init_addr:","sourceCodeStart":658,"sourceCodeEnd":694,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/entrypoints/engine.py#L658-L694","documentation":"When nnodes > 1 and weight-cache daemons are enabled, every node's daemon must join one shared rendezvous process group. Without --dist-init-addr each node defaults to its own 127.0.0.1 endpoint and the joint group can never form, so the engine refuses to launch at subprocess-startup time with ValueError.","triggerScenarios":"Launching sgl.Engine (or _launch_subprocesses) with server_args.nnodes > 1, a weight-cache mode that spawns daemons, and dist_init_addr unset/empty.","commonSituations":"Multi-node launch scripts (torchrun/slurm) that relied on single-node defaults; adding --nnodes 2 to a previously single-node config without adding the rendezvous address; empty-string dist-init-addr from templated YAML.","solutions":["Pass --dist-init-addr host:port pointing at the node-rank-0 reachable address, e.g. --dist-init-addr 10.0.0.1:5000.","If the template/launcher (e.g. slurm launcher) is supposed to inject it, verify the variable actually reaches server_args (print server_args.resolved_dict()).","Fall back to single node (nnodes=1) if multi-node weight caching isn't actually needed."],"exampleFix":"# before\nengine = sgl.Engine(model_path=..., nnodes=2, node_rank=0,\n                    weight_cache_mode=\"daemon\")  # ValueError\n\n# after\nengine = sgl.Engine(model_path=..., nnodes=2, node_rank=0,\n                    dist_init_addr=\"10.0.0.1:5000\",\n                    weight_cache_mode=\"daemon\")","handlingStrategy":"validation","validationCode":"sa = dict(model_path=..., nnodes=2, node_rank=0,\n            dist_init_addr=\"10.0.0.1:5000\")  # must be set when nnodes>1\nengine = sgl.Engine(**sa)\nassert not (sa.get(\"nnodes\", 1) > 1) or sa.get(\"dist_init_addr\")","typeGuard":null,"tryCatchPattern":"try:\n    engine = sgl.Engine(**kwargs)\nexcept ValueError as e:\n    if \"dist-init-addr\" in str(e):\n        kwargs[\"dist_init_addr\"] = f\"{my_node0_ip}:5000\"\n        engine = sgl.Engine(**kwargs)\n    else:\n        raise","preventionTips":["Template multi-node launches to always emit --dist-init-addr alongside --nnodes.","Use the node-rank-0 primary IP, not localhost, in multi-node setups.","Log server_args.resolved_dict() at launch to confirm dist_init_addr arrived."],"tags":["sglang","multi-node","distributed","weight-cache","startup"],"backgroundTag":"missing-dist-init-addr","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}