{"record":{"id":"45b4fbf53fd328be","repo":"sgl-project/sglang","slug":"weight-cache-daemon-for-rank-global-rank-is-alre","errorCode":null,"errorMessage":"Weight cache daemon for rank {global_rank} is already running (pid={pid}, ready={ready_path}). Stop the existing daemon before launching a new one, or pass force=True (--force) to kill it and take over.","messagePattern":"Weight cache daemon for rank (.+?) is already running \\(pid=(.+?), ready=(.+?)\\)\\. Stop the existing daemon before launching a new one, or pass force=True \\(--force\\) to kill it and take over\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/weight_cache/protocol.py","lineNumber":382,"sourceCode":"\n    If the .ready file exists and the recorded PID is still alive, the daemon\n    is still running — raise RuntimeError so the caller doesn't clobber it,\n    unless ``force`` is set, in which case the running daemon is killed and its\n    files are taken over (stale-takeover path for a wedged/orphaned daemon).\n    If the PID is dead (or unreadable), the files are stale leftovers from a\n    crashed/killed daemon and are safe to remove.\n    \"\"\"\n    ready_path = get_ready_path(global_rank)\n    socket_path = get_socket_path(global_rank)\n\n    if not os.path.exists(ready_path) and not os.path.exists(socket_path):\n        return\n\n    pid = _read_ready_pid(ready_path) if os.path.exists(ready_path) else None\n\n    if pid is not None and _is_pid_alive(pid):\n        if not force:\n            raise RuntimeError(\n                f\"Weight cache daemon for rank {global_rank} is already running \"\n                f\"(pid={pid}, ready={ready_path}). Stop the existing daemon before \"\n                f\"launching a new one, or pass force=True (--force) to kill it and \"\n                f\"take over.\"\n            )\n        logger.warning(\n            f\"[weight_cache] force takeover: killing existing daemon pid={pid} \"\n            f\"for rank {global_rank} and reclaiming its socket/ready files.\"\n        )\n        try:\n            os.kill(pid, signal.SIGKILL)\n        except ProcessLookupError:\n            pass\n\n    for path in (ready_path, socket_path):\n        if os.path.exists(path):\n            os.unlink(path)\n            logger.info(f\"Removed stale daemon file: {path}\")","sourceCodeStart":364,"sourceCodeEnd":400,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/weight_cache/protocol.py#L364-L400","documentation":"cleanup_stale_daemon_files found a ready file whose pid is alive, so a weight cache daemon for this rank already exists. Refuses to launch unless force=True to avoid two daemons fighting over the same socket/shard.","triggerScenarios":"Relaunching a server while the previous weight cache daemon survived; passing force=False (default) to launch_weight_cache_daemons.","commonSituations":"Server killed without cleanup (SIGKILL), daemon orphaned; rapid restart loops in CI.","solutions":["Stop the existing daemon (kill the pid in the ready file) then relaunch","Pass --force / force=True to kill it and take over","Remove stale ready/socket files after confirming the pid is dead"],"exampleFix":"# before\nlaunch_weight_cache_daemons(args)\n# after\nlaunch_weight_cache_daemons(args, force=True)","handlingStrategy":"fallback","validationCode":"pid = _read_ready_pid(ready_path)\nif pid and _is_pid_alive(pid) and not force:\n    os.kill(pid, signal.SIGTERM)","typeGuard":null,"tryCatchPattern":"try:\n    launch_weight_cache_daemons(args)\nexcept RuntimeError as e:\n    if 'already running' in str(e):\n        launch_weight_cache_daemons(args, force=True)","preventionTips":["Pass --force in restart scripts","Clean up daemon files in teardown handlers"],"tags":["daemon","weight-cache","lifecycle","stale-process"],"backgroundTag":"daemon-already-running","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}