sgl-project/sglang · error · ValueError

Multi-node weight cache daemons (nnodes > 1) require --dist-

Error message

Multi-node weight cache daemons (nnodes > 1) require --dist-init-addr so all nodes rendezvous at the same endpoint.

What it means

When nnodes > 1 and weight-cache daemons are enabled, every node's daemon must join one shared rendezvous process group. Without --dist-init-addr each node defaults to its own 127.0.0.1 endpoint and the joint group can never form, so the engine refuses to launch at subprocess-startup time with ValueError.

Source

Thrown at python/sglang/srt/entrypoints/engine.py:676

        All daemon processes join the same NCCL distributed group so that
        TP-sharded model loading works correctly. Each daemon holds its
        rank's weight shard in GPU memory and serves IPC handles.

        Lifecycle: these daemons are *co-terminal* with the engine. They are
        children of this process (kill_itself_when_parent_died installs
        PR_SET_PDEATHSIG) and are gracefully reaped in ``shutdown()``. They do
        NOT persist across engine restarts, so ``--weight-cache-mode daemon``
        on its own does not deliver a faster restart -- the first start is in
        fact slower (disk-load into the daemon plus the IPC handshake). The
        fast-recovery story is the standalone launcher
        (``python -m sglang.srt.weight_cache.daemon``) plus
        ``--weight-cache-mode client``, where the daemon outlives the engine.
        """
        # Multi-node needs an explicit rendezvous address; otherwise each node
        # picks its own local 127.0.0.1 port (below) and the per-node daemons
        # can never form the joint process group.
        if server_args.nnodes > 1 and not server_args.dist_init_addr:
            raise ValueError(
                "Multi-node weight cache daemons (nnodes > 1) require "
                "--dist-init-addr so all nodes rendezvous at the same endpoint."
            )

        tp_size = server_args.tp_size

        pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = (
            _calculate_rank_ranges(
                server_args.nnodes,
                get_parallel().pp_size,
                tp_size,
                server_args.node_rank,
            )
        )

        # Build the distributed init method (multi-node uses the user-provided
        # dist_init_addr so all nodes reach the same endpoint).
        if server_args.dist_init_addr:

View on GitHub (pinned to 0132848349)

Solutions

1. Pass --dist-init-addr host:port pointing at the node-rank-0 reachable address, e.g. --dist-init-addr 10.0.0.1:5000.
2. If the template/launcher (e.g. slurm launcher) is supposed to inject it, verify the variable actually reaches server_args (print server_args.resolved_dict()).
3. Fall back to single node (nnodes=1) if multi-node weight caching isn't actually needed.

Example fix

# before
engine = sgl.Engine(model_path=..., nnodes=2, node_rank=0,
                    weight_cache_mode="daemon")  # ValueError

# after
engine = sgl.Engine(model_path=..., nnodes=2, node_rank=0,
                    dist_init_addr="10.0.0.1:5000",
                    weight_cache_mode="daemon")

sa = dict(model_path=..., nnodes=2, node_rank=0,
            dist_init_addr="10.0.0.1:5000")  # must be set when nnodes>1
engine = sgl.Engine(**sa)
assert not (sa.get("nnodes", 1) > 1) or sa.get("dist_init_addr")

try:
    engine = sgl.Engine(**kwargs)
except ValueError as e:
    if "dist-init-addr" in str(e):
        kwargs["dist_init_addr"] = f"{my_node0_ip}:5000"
        engine = sgl.Engine(**kwargs)
    else:
        raise

When it happens

Trigger: Launching sgl.Engine (or _launch_subprocesses) with server_args.nnodes > 1, a weight-cache mode that spawns daemons, and dist_init_addr unset/empty.

Common situations: Multi-node launch scripts (torchrun/slurm) that relied on single-node defaults; adding --nnodes 2 to a previously single-node config without adding the rendezvous address; empty-string dist-init-addr from templated YAML.

Related errors

• Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank}
• Weight cache daemon (pid={p.pid}) exited prematurely with co
• --mm-feature-transport=cuda_ipc only supports a single node.
• Waiting for main node timeout!
• world_size ({world_size}) is less than tensor_parallel_degre

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8805ea3d1cce2913. Report an issue: GitHub.