sgl-project/sglang · critical · RuntimeError

[IpcModelLoader] Error communicating with daemon at {self.so

Error message

[IpcModelLoader] Error communicating with daemon at {self.socket_path}: {e}

What it means

[IpcModelLoader] Error communicating with daemon at {self.socket_path}: {e}

Source

Thrown at python/sglang/srt/weight_cache/ipc_loader.py:573

            result = recv_msg(sock)

            if result.get("status") != "ok":
                daemon_config = result.get("daemon_config", {})
                raise RuntimeError(
                    f"[IpcModelLoader] Daemon config mismatch!\n"
                    f"  Engine config: {engine_config.to_dict()}\n"
                    f"  Daemon config: {daemon_config}"
                )

            backend_name = result.get("transport_backend", TORCH_IPC_BACKEND)
            self._transport_backend = get_client_transport_backend(backend_name)
            result = self._transport_backend.recv_fetch_state_response(sock, result)
            return result

        except RuntimeError:
            raise
        except Exception as e:
            raise RuntimeError(
                f"[IpcModelLoader] Error communicating with daemon at "
                f"{self.socket_path}: {e}"
            ) from e
        finally:
            sock.close()

    def _fallback_load(self, model_config, device_config) -> nn.Module:
        """Fall back to DefaultModelLoader for disk-based loading."""
        from sglang.srt.configs.load_config import LoadConfig
        from sglang.srt.model_loader.loader import DefaultModelLoader

        fallback_config = LoadConfig(
            load_format=self._fallback_load_format,
            download_dir=self.load_config.download_dir,
            model_loader_extra_config=self.load_config.model_loader_extra_config,
            tp_rank=self.load_config.tp_rank,
        )
        loader_cls = self._fallback_loader_cls or DefaultModelLoader

View on GitHub (pinned to 0132848349)

Solutions

1. Check that the weight cache daemon for the rank is alive (ready file pid) and restart it, or relaunch with --force
2. Inspect the wrapped cause (__cause__) for the underlying socket/timeout error
3. Disable the cache (--weight-cache-mode off) to unblock while debugging

Example fix

# before
model = loader.load_model(weights_info)
# after
try:
    model = loader.load_model(weights_info)
except RuntimeError as e:
    if "IpcModelLoader" in str(e):
        logger.error("daemon down: %s", e.__cause__)
        raise

pid = _read_ready_pid(ready_path)
assert pid is not None and _is_pid_alive(pid), 'daemon not alive'

try:
    loader.load_model(...)
except RuntimeError as e:
    if '[IpcModelLoader]' in str(e):
        handle_daemon_down(e.__cause__)

When it happens

Trigger: Calling load_model via IpcModelLoader when the weight cache daemon has died, its unix socket was removed, or any non-RuntimeError exception (timeout, socket error, bad message) escapes _fetch_from_cache.

Common situations: Weight cache daemon crashed OOM after startup; socket path mismatch between ranks; daemon killed manually while server loading weights.

Related errors

• Weight cache daemon for pp_rank={pp_rank} tp_rank={tp_rank}
• CUDA VMM POSIX FD broker returned no file descriptor
• [weight_cache:{where}] quantization method {quant_method!r}
• Connection closed while reading message header
• Weight cache daemon for rank {global_rank} is already runnin

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0b8d85a56b9455c2. Report an issue: GitHub.