sgl-project/sglang · critical · RuntimeError
[IpcModelLoader] Error communicating with daemon at
Error message
[IpcModelLoader] Error communicating with daemon at {self.socket_path}: {e} What it means
IpcModelLoader error wrapper: communication with the weight-cache daemon over the Unix socket at the given path failed (exception 'e' in the message). Means the daemon is not running, the socket path is wrong, or the connection broke mid-protocol.
Solutions
- Check that the weight cache daemon for the rank is alive (ready file pid) and restart it, or relaunch with --force
- Inspect the wrapped cause (__cause__) for the underlying socket/timeout error
- Disable the cache (--weight-cache-mode off) to unblock while debugging
Example fix
# before
model = loader.load_model(weights_info)
# after
try:
model = loader.load_model(weights_info)
except RuntimeError as e:
if "IpcModelLoader" in str(e):
logger.error("daemon down: %s", e.__cause__)
raise Defensive patterns
Strategy: try-catch
Validate before calling
pid = _read_ready_pid(ready_path) assert pid is not None and _is_pid_alive(pid), 'daemon not alive'
Try / catch
try:
loader.load_model(...)
except RuntimeError as e:
if '[IpcModelLoader]' in str(e):
handle_daemon_down(e.__cause__) Prevention
- Ensure daemons are launched and ready before starting rank processes
- Monitor daemon liveness; use --force on restart
When it happens
Trigger: Calling load_model via IpcModelLoader when the weight cache daemon has died, its unix socket was removed, or any non-RuntimeError exception (timeout, socket error, bad message) escapes _fetch_from_cache.
Common situations: Weight cache daemon crashed OOM after startup; socket path mismatch between ranks; daemon killed manually while server loading weights.
Related errors
- Connection closed while reading message header
- CUDA VMM POSIX FD broker returned no file descriptor
- sendmsg sent bytes, expected
- Weight cache daemon for pp_rank=
- Weight cache daemon for rank
AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28).
Data as JSON: /api/errors/0b8d85a56b9455c2.
Report an issue: GitHub.
Appendix: source
Thrown at python/sglang/srt/weight_cache/ipc_loader.py:573
result = recv_msg(sock)
if result.get("status") != "ok":
daemon_config = result.get("daemon_config", {})
raise RuntimeError(
f"[IpcModelLoader] Daemon config mismatch!\n"
f" Engine config: {engine_config.to_dict()}\n"
f" Daemon config: {daemon_config}"
)
backend_name = result.get("transport_backend", TORCH_IPC_BACKEND)
self._transport_backend = get_client_transport_backend(backend_name)
result = self._transport_backend.recv_fetch_state_response(sock, result)
return result
except RuntimeError:
raise
except Exception as e:
raise RuntimeError(
f"[IpcModelLoader] Error communicating with daemon at "
f"{self.socket_path}: {e}"
) from e
finally:
sock.close()
def _fallback_load(self, model_config, device_config) -> nn.Module:
"""Fall back to DefaultModelLoader for disk-based loading."""
from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.model_loader.loader import DefaultModelLoader
fallback_config = LoadConfig(
load_format=self._fallback_load_format,
download_dir=self.load_config.download_dir,
model_loader_extra_config=self.load_config.model_loader_extra_config,
tp_rank=self.load_config.tp_rank,
)
loader_cls = self._fallback_loader_cls or DefaultModelLoaderView on GitHub (pinned to 0132848349)