{"record":{"id":"0b8d85a56b9455c2","repo":"sgl-project/sglang","slug":"ipcmodelloader-error-communicating-with-daemon-a","errorCode":null,"errorMessage":"[IpcModelLoader] Error communicating with daemon at {self.socket_path}: {e}","messagePattern":"\\[IpcModelLoader\\] Error communicating with daemon at (.+?): (.+?)","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/weight_cache/ipc_loader.py","lineNumber":573,"sourceCode":"            result = recv_msg(sock)\n\n            if result.get(\"status\") != \"ok\":\n                daemon_config = result.get(\"daemon_config\", {})\n                raise RuntimeError(\n                    f\"[IpcModelLoader] Daemon config mismatch!\\n\"\n                    f\"  Engine config: {engine_config.to_dict()}\\n\"\n                    f\"  Daemon config: {daemon_config}\"\n                )\n\n            backend_name = result.get(\"transport_backend\", TORCH_IPC_BACKEND)\n            self._transport_backend = get_client_transport_backend(backend_name)\n            result = self._transport_backend.recv_fetch_state_response(sock, result)\n            return result\n\n        except RuntimeError:\n            raise\n        except Exception as e:\n            raise RuntimeError(\n                f\"[IpcModelLoader] Error communicating with daemon at \"\n                f\"{self.socket_path}: {e}\"\n            ) from e\n        finally:\n            sock.close()\n\n    def _fallback_load(self, model_config, device_config) -> nn.Module:\n        \"\"\"Fall back to DefaultModelLoader for disk-based loading.\"\"\"\n        from sglang.srt.configs.load_config import LoadConfig\n        from sglang.srt.model_loader.loader import DefaultModelLoader\n\n        fallback_config = LoadConfig(\n            load_format=self._fallback_load_format,\n            download_dir=self.load_config.download_dir,\n            model_loader_extra_config=self.load_config.model_loader_extra_config,\n            tp_rank=self.load_config.tp_rank,\n        )\n        loader_cls = self._fallback_loader_cls or DefaultModelLoader","sourceCodeStart":555,"sourceCodeEnd":591,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/weight_cache/ipc_loader.py#L555-L591","documentation":"[IpcModelLoader] Error communicating with daemon at {self.socket_path}: {e}","triggerScenarios":"Calling load_model via IpcModelLoader when the weight cache daemon has died, its unix socket was removed, or any non-RuntimeError exception (timeout, socket error, bad message) escapes _fetch_from_cache.","commonSituations":"Weight cache daemon crashed OOM after startup; socket path mismatch between ranks; daemon killed manually while server loading weights.","solutions":["Check that the weight cache daemon for the rank is alive (ready file pid) and restart it, or relaunch with --force","Inspect the wrapped cause (__cause__) for the underlying socket/timeout error","Disable the cache (--weight-cache-mode off) to unblock while debugging"],"exampleFix":"# before\nmodel = loader.load_model(weights_info)\n# after\ntry:\n    model = loader.load_model(weights_info)\nexcept RuntimeError as e:\n    if \"IpcModelLoader\" in str(e):\n        logger.error(\"daemon down: %s\", e.__cause__)\n        raise","handlingStrategy":"try-catch","validationCode":"pid = _read_ready_pid(ready_path)\nassert pid is not None and _is_pid_alive(pid), 'daemon not alive'","typeGuard":null,"tryCatchPattern":"try:\n    loader.load_model(...)\nexcept RuntimeError as e:\n    if '[IpcModelLoader]' in str(e):\n        handle_daemon_down(e.__cause__)","preventionTips":["Ensure daemons are launched and ready before starting rank processes","Monitor daemon liveness; use --force on restart"],"tags":["weight-cache","ipc","daemon","socket"],"backgroundTag":"daemon-connection-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T11:17:15.048Z"}