What causes "Timed out waiting for ACK from FlexKV layerwise worker" in sgl-project/sglang?

Inspect the layerwise worker's stack (py-spy/gdb) to find where it stalled during eventfd registration

sgl-project/sglang · critical · RuntimeError

Timed out waiting for ACK from FlexKV layerwise worker

Error message

Timed out waiting for ACK from FlexKV layerwise worker

What it means

After sending eventfds for a layer_done_counter to the worker over the socket, the connector waits 30s (sock.settimeout(30.0)) for a 1-byte ACK. A socket.timeout is converted to RuntimeError: the worker accepted the connection but never responded in time.

Source

Thrown at python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py:894

                # Phase 2: send 16-byte metadata + per-counter FDs + read ACK.
                num_counters = self.layer_done_counter.num_counters
                metadata = struct.pack(
                    "iiii",
                    self.rank_info.tp_rank_per_node,
                    self.model_config.tp_size_per_node,
                    self.rank_info.num_layers_per_pp_stage,
                    num_counters,
                )
                sock.sendall(metadata)
                for counter_id in range(num_counters):
                    fds = self.layer_done_counter.events[counter_id].load_event_fds
                    send_fds(sock, fds, struct.pack("i", counter_id))
                sock.settimeout(30.0)
                try:
                    ack = sock.recv(1)
                except socket.timeout as exc:
                    raise RuntimeError(
                        "Timed out waiting for ACK from FlexKV layerwise worker"
                    ) from exc
                if not ack or ack[0] != 1:
                    raise RuntimeError(
                        f"FlexKV layerwise worker NACK'd eventfd transfer "
                        f"(ack={ack!r})"
                    )
                logger.info(
                    "[FlexKV] Eventfd handshake complete %s counters=%d layers=%d",
                    self._label,
                    num_counters,
                    self.rank_info.num_layers_per_pp_stage,
                )
                return
            except Exception as exc:  # noqa: BLE001
                last_error = exc
                logger.warning(
                    "[FlexKV] Eventfd handshake send_attempt=%d/%d failed: %s",

View on GitHub (pinned to 0132848349)

Solutions

Inspect the layerwise worker's stack (py-spy/gdb) to find where it stalled during eventfd registration
Check for worker crash logs / core dumps at the same timestamp; restart worker and connector together
If load-related, raise the 30s ACK timeout or reduce the number of counters registered per connection

Defensive patterns

Strategy: retry

Try / catch

for attempt in range(3):
    try:
        connector = FlexKVConnector(...)
        break
    except RuntimeError as e:
        if 'Timed out waiting for ACK' in str(e) and attempt < 2:
            logger.warning('ACK timeout, restarting worker and retrying')
            restart_flexkv_worker()
            continue
        raise

Prevention

Keep the worker's logs correlated by timestamp so ACK timeouts can be matched to worker stalls
Cap concurrent counter registrations during init to bound ACK latency below the 30s timeout

When it happens

Trigger: The worker received fds but is blocked/hung (e.g. stuck in a CUDA or FlexKV call), its eventfd registration path deadlocked, or the machine is heavily loaded so the ACK is delayed past 30s.

Common situations: Worker hang due to a driver/FlexKV internal error; oversized batch of counters making registration slow; transient extreme load during startup; worker crashed after accept but before recv/send.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/15be076a5113435f. Report an issue: GitHub.