sgl-project/sglang · error · RuntimeError

Weight cache daemon for rank {global_rank} is already runnin

Error message

Weight cache daemon for rank {global_rank} is already running (pid={pid}, ready={ready_path}). Stop the existing daemon before launching a new one, or pass force=True (--force) to kill it and take over.

What it means

cleanup_stale_daemon_files found a ready file whose pid is alive, so a weight cache daemon for this rank already exists. Refuses to launch unless force=True to avoid two daemons fighting over the same socket/shard.

Source

Thrown at python/sglang/srt/weight_cache/protocol.py:382

    If the .ready file exists and the recorded PID is still alive, the daemon
    is still running — raise RuntimeError so the caller doesn't clobber it,
    unless ``force`` is set, in which case the running daemon is killed and its
    files are taken over (stale-takeover path for a wedged/orphaned daemon).
    If the PID is dead (or unreadable), the files are stale leftovers from a
    crashed/killed daemon and are safe to remove.
    """
    ready_path = get_ready_path(global_rank)
    socket_path = get_socket_path(global_rank)

    if not os.path.exists(ready_path) and not os.path.exists(socket_path):
        return

    pid = _read_ready_pid(ready_path) if os.path.exists(ready_path) else None

    if pid is not None and _is_pid_alive(pid):
        if not force:
            raise RuntimeError(
                f"Weight cache daemon for rank {global_rank} is already running "
                f"(pid={pid}, ready={ready_path}). Stop the existing daemon before "
                f"launching a new one, or pass force=True (--force) to kill it and "
                f"take over."
            )
        logger.warning(
            f"[weight_cache] force takeover: killing existing daemon pid={pid} "
            f"for rank {global_rank} and reclaiming its socket/ready files."
        )
        try:
            os.kill(pid, signal.SIGKILL)
        except ProcessLookupError:
            pass

    for path in (ready_path, socket_path):
        if os.path.exists(path):
            os.unlink(path)
            logger.info(f"Removed stale daemon file: {path}")

View on GitHub (pinned to 0132848349)

Solutions

1. Stop the existing daemon (kill the pid in the ready file) then relaunch
2. Pass --force / force=True to kill it and take over
3. Remove stale ready/socket files after confirming the pid is dead

Example fix

# before
launch_weight_cache_daemons(args)
# after
launch_weight_cache_daemons(args, force=True)

pid = _read_ready_pid(ready_path)
if pid and _is_pid_alive(pid) and not force:
    os.kill(pid, signal.SIGTERM)

try:
    launch_weight_cache_daemons(args)
except RuntimeError as e:
    if 'already running' in str(e):
        launch_weight_cache_daemons(args, force=True)

When it happens

Trigger: Relaunching a server while the previous weight cache daemon survived; passing force=False (default) to launch_weight_cache_daemons.

Common situations: Server killed without cleanup (SIGKILL), daemon orphaned; rapid restart loops in CI.

Related errors

• Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank}
• [IpcModelLoader] Error communicating with daemon at {self.so
• Encoder request was released: {state.req_id}
• Multi-node weight cache daemons (nnodes > 1) require --dist-
• Weight cache daemon (pid={p.pid}) exited prematurely with co

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/45b4fbf53fd328be. Report an issue: GitHub.