sgl-project/sglang · critical · RuntimeError

[FlexKV] Failed to connect to eventfd socket {self._layerwis

Error message

[FlexKV] Failed to connect to eventfd socket {self._layerwise_socket} after {max_retries} attempts

What it means

During connector init, _send_eventfds_to_worker retries connecting a Unix socket to the FlexKV layerwise worker (self._layerwise_socket) up to max_retries with a retry interval. If the worker process never starts listening, the final FileNotFoundError/ConnectionRefusedError is re-raised as RuntimeError.

Source

Thrown at python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py:870

            sock: Optional[socket.socket] = None
            try:
                # Phase 1: connect (worker may not yet be up).
                for attempt in range(max_retries):
                    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
                    try:
                        sock.connect(self._layerwise_socket)
                        logger.info(
                            "[FlexKV] Eventfd connected %s socket=%s attempts=%d",
                            self._label,
                            self._layerwise_socket,
                            attempt + 1,
                        )
                        break
                    except (FileNotFoundError, ConnectionRefusedError) as exc:
                        sock.close()
                        sock = None
                        if attempt == max_retries - 1:
                            raise RuntimeError(
                                f"[FlexKV] Failed to connect to eventfd socket "
                                f"{self._layerwise_socket} after {max_retries} attempts"
                            ) from exc
                        time.sleep(retry_interval)
                assert sock is not None

                # Phase 2: send 16-byte metadata + per-counter FDs + read ACK.
                num_counters = self.layer_done_counter.num_counters
                metadata = struct.pack(
                    "iiii",
                    self.rank_info.tp_rank_per_node,
                    self.model_config.tp_size_per_node,
                    self.rank_info.num_layers_per_pp_stage,
                    num_counters,
                )
                sock.sendall(metadata)
                for counter_id in range(num_counters):
                    fds = self.layer_done_counter.events[counter_id].load_event_fds

View on GitHub (pinned to 0132848349)

Solutions

Verify the FlexKV layerwise worker process is running and check its log for early crashes
Confirm self._layerwise_socket matches the path the worker actually binds (shared volume /tmp on the same host, not cross-node)
Increase max_retries/retry_interval or add a readiness gate so scheduler init waits for the socket to appear

Defensive patterns

Strategy: retry

Validate before calling

import os, socket

def flexkv_socket_ready(path: str) -> bool:
    if not os.path.exists(path):
        return False
    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    try:
        s.connect(path)
        return True
    except OSError:
        return False
    finally:
        s.close()

Try / catch

try:
    connector = FlexKVConnector(...)
except RuntimeError as e:
    if 'Failed to connect to eventfd socket' in str(e):
        logger.warning('layerwise worker not up; retrying init')
        time.sleep(5)
        connector = FlexKVConnector(...)
    else:
        raise

Prevention

Start and health-check the layerwise worker before launching the scheduler
Verify the socket path is on a filesystem shared by exactly those two processes on the same host

When it happens

Trigger: FlexKV connector constructed on a node where the layerwise worker daemon is not up yet (slow start), was never launched, or listens at a different socket path; also when the worker crashed immediately on startup.

Common situations: Wrong layerwise_socket path configuration on multi-node setups; the worker container/pod failing before the scheduler connects; races at server startup where init proceeds before the worker binds its socket.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/701427bdefc2a24d. Report an issue: GitHub.