{"record":{"id":"701427bdefc2a24d","repo":"sgl-project/sglang","slug":"flexkv-failed-to-connect-to-eventfd-socket-self","errorCode":null,"errorMessage":"[FlexKV] Failed to connect to eventfd socket {self._layerwise_socket} after {max_retries} attempts","messagePattern":"\\[FlexKV\\] Failed to connect to eventfd socket (.+?) after (.+?) attempts","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py","lineNumber":870,"sourceCode":"            sock: Optional[socket.socket] = None\n            try:\n                # Phase 1: connect (worker may not yet be up).\n                for attempt in range(max_retries):\n                    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n                    try:\n                        sock.connect(self._layerwise_socket)\n                        logger.info(\n                            \"[FlexKV] Eventfd connected %s socket=%s attempts=%d\",\n                            self._label,\n                            self._layerwise_socket,\n                            attempt + 1,\n                        )\n                        break\n                    except (FileNotFoundError, ConnectionRefusedError) as exc:\n                        sock.close()\n                        sock = None\n                        if attempt == max_retries - 1:\n                            raise RuntimeError(\n                                f\"[FlexKV] Failed to connect to eventfd socket \"\n                                f\"{self._layerwise_socket} after {max_retries} attempts\"\n                            ) from exc\n                        time.sleep(retry_interval)\n                assert sock is not None\n\n                # Phase 2: send 16-byte metadata + per-counter FDs + read ACK.\n                num_counters = self.layer_done_counter.num_counters\n                metadata = struct.pack(\n                    \"iiii\",\n                    self.rank_info.tp_rank_per_node,\n                    self.model_config.tp_size_per_node,\n                    self.rank_info.num_layers_per_pp_stage,\n                    num_counters,\n                )\n                sock.sendall(metadata)\n                for counter_id in range(num_counters):\n                    fds = self.layer_done_counter.events[counter_id].load_event_fds","sourceCodeStart":852,"sourceCodeEnd":888,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py#L852-L888","documentation":"During connector init, _send_eventfds_to_worker retries connecting a Unix socket to the FlexKV layerwise worker (self._layerwise_socket) up to max_retries with a retry interval. If the worker process never starts listening, the final FileNotFoundError/ConnectionRefusedError is re-raised as RuntimeError.","triggerScenarios":"FlexKV connector constructed on a node where the layerwise worker daemon is not up yet (slow start), was never launched, or listens at a different socket path; also when the worker crashed immediately on startup.","commonSituations":"Wrong layerwise_socket path configuration on multi-node setups; the worker container/pod failing before the scheduler connects; races at server startup where init proceeds before the worker binds its socket.","solutions":["Verify the FlexKV layerwise worker process is running and check its log for early crashes","Confirm self._layerwise_socket matches the path the worker actually binds (shared volume /tmp on the same host, not cross-node)","Increase max_retries/retry_interval or add a readiness gate so scheduler init waits for the socket to appear"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import os, socket\n\ndef flexkv_socket_ready(path: str) -> bool:\n    if not os.path.exists(path):\n        return False\n    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n    try:\n        s.connect(path)\n        return True\n    except OSError:\n        return False\n    finally:\n        s.close()","typeGuard":null,"tryCatchPattern":"try:\n    connector = FlexKVConnector(...)\nexcept RuntimeError as e:\n    if 'Failed to connect to eventfd socket' in str(e):\n        logger.warning('layerwise worker not up; retrying init')\n        time.sleep(5)\n        connector = FlexKVConnector(...)\n    else:\n        raise","preventionTips":["Start and health-check the layerwise worker before launching the scheduler","Verify the socket path is on a filesystem shared by exactly those two processes on the same host"],"tags":["flexkv","unix-socket","worker-startup","connection-refused"],"backgroundTag":"socket-connection-refused","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}