{"record":{"id":"0b8d85a56b9455c2","repo":"sgl-project/sglang","slug":"ipcmodelloader-error-communicating-with-daemon-a","errorCode":null,"errorMessage":"[IpcModelLoader] Error communicating with daemon at {self.socket_path}: {e}","messagePattern":"\\[IpcModelLoader\\] Error communicating with daemon at (.+?): (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/weight_cache/ipc_loader.py","lineNumber":573,"sourceCode":"            result = recv_msg(sock)\n\n            if result.get(\"status\") != \"ok\":\n                daemon_config = result.get(\"daemon_config\", {})\n                raise RuntimeError(\n                    f\"[IpcModelLoader] Daemon config mismatch!\\n\"\n                    f\"  Engine config: {engine_config.to_dict()}\\n\"\n                    f\"  Daemon config: {daemon_config}\"\n                )\n\n            backend_name = result.get(\"transport_backend\", TORCH_IPC_BACKEND)\n            self._transport_backend = get_client_transport_backend(backend_name)\n            result = self._transport_backend.recv_fetch_state_response(sock, result)\n            return result\n\n        except RuntimeError:\n            raise\n        except Exception as e:\n            raise RuntimeError(\n                f\"[IpcModelLoader] Error communicating with daemon at \"\n                f\"{self.socket_path}: {e}\"\n            ) from e\n        finally:\n            sock.close()\n\n    def _fallback_load(self, model_config, device_config) -> nn.Module:\n        \"\"\"Fall back to DefaultModelLoader for disk-based loading.\"\"\"\n        from sglang.srt.configs.load_config import LoadConfig\n        from sglang.srt.model_loader.loader import DefaultModelLoader\n\n        fallback_config = LoadConfig(\n            load_format=self._fallback_load_format,\n            download_dir=self.load_config.download_dir,\n            model_loader_extra_config=self.load_config.model_loader_extra_config,\n            tp_rank=self.load_config.tp_rank,\n        )\n        loader_cls = self._fallback_loader_cls or DefaultModelLoader","sourceCodeStart":555,"sourceCodeEnd":591,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/weight_cache/ipc_loader.py#L555-L591","documentation":"IpcModelLoader error wrapper: communication with the weight-cache daemon over the Unix socket at the given path failed (exception 'e' in the message). Means the daemon is not running, the socket path is wrong, or the connection broke mid-protocol.","triggerScenarios":"Calling load_model via IpcModelLoader when the weight cache daemon has died, its unix socket was removed, or any non-RuntimeError exception (timeout, socket error, bad message) escapes _fetch_from_cache.","commonSituations":"Weight cache daemon crashed OOM after startup; socket path mismatch between ranks; daemon killed manually while server loading weights.","solutions":["Check that the weight cache daemon for the rank is alive (ready file pid) and restart it, or relaunch with --force","Inspect the wrapped cause (__cause__) for the underlying socket/timeout error","Disable the cache (--weight-cache-mode off) to unblock while debugging"],"exampleFix":"# before\nmodel = loader.load_model(weights_info)\n# after\ntry:\n    model = loader.load_model(weights_info)\nexcept RuntimeError as e:\n    if \"IpcModelLoader\" in str(e):\n        logger.error(\"daemon down: %s\", e.__cause__)\n        raise","handlingStrategy":"try-catch","validationCode":"pid = _read_ready_pid(ready_path)\nassert pid is not None and _is_pid_alive(pid), 'daemon not alive'","typeGuard":null,"tryCatchPattern":"try:\n    loader.load_model(...)\nexcept RuntimeError as e:\n    if '[IpcModelLoader]' in str(e):\n        handle_daemon_down(e.__cause__)","preventionTips":["Ensure daemons are launched and ready before starting rank processes","Monitor daemon liveness; use --force on restart"],"tags":["weight-cache","ipc","daemon","socket"],"backgroundTag":"daemon-connection-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","contentChangedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}