sgl-project/sglang · critical · RuntimeError

[FlexKV] Failed to send eventfds to {self._layerwise_socket}

Error message

[FlexKV] Failed to send eventfds to {self._layerwise_socket} after {max_send_retries} attempts: {last_error}

What it means

_send_eventfds_to_worker wraps the whole send phase in max_send_retries; any per-attempt exception (raise ... in the loop body) is retried after closing the socket. If every attempt fails, the last error is re-raised with this aggregated RuntimeError naming the socket path and retry count.

Source

Thrown at python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py:922

                    self._label,
                    num_counters,
                    self.rank_info.num_layers_per_pp_stage,
                )
                return
            except Exception as exc:  # noqa: BLE001
                last_error = exc
                logger.warning(
                    "[FlexKV] Eventfd handshake send_attempt=%d/%d failed: %s",
                    send_attempt + 1,
                    max_send_retries,
                    exc,
                )
            finally:
                if sock is not None:
                    sock.close()
                time.sleep(retry_interval)

        raise RuntimeError(
            f"[FlexKV] Failed to send eventfds to {self._layerwise_socket} "
            f"after {max_send_retries} attempts: {last_error}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Read last_error context in the log: broken pipe points to a dead worker, KeyError on events[...] to an unregistered counter
2. Reproduce with a single counter to isolate whether the failure is per-counter or per-connection
3. Fix the root cause on the worker (crash/registration), then restart both sides; increasing retries alone rarely helps here

try:
    _send_eventfds_to_worker_wrapper(...)
except RuntimeError as e:
    if 'Failed to send eventfds' in str(e):
        capture_last_error_from_log(e)  # inspect nested cause for broken pipe vs KeyError
        restart_flexkv_worker()
        raise

When it happens

Trigger: Persistent send-phase failures: send_fds raising (broken pipe because the worker died mid-handshake), counter_id missing from layer_done_counter.events, or struct/serialization errors — repeated across all max_send_retries attempts.

Common situations: Worker crashing between connect and send; referencing an unregistered counter_id; repeated transient socket errors masking an underlying worker bug.

Related errors

• [FlexKV] Failed to connect to eventfd socket {self._layerwis
• Timed out waiting for ACK from FlexKV layerwise worker
• FlexKV layerwise worker NACK'd eventfd transfer (ack={ack!r}
• Unsupported KV cache type {type(kvcache).__name__}: expected
• Tag mismatch: expected CMD_LAYERWISE, got {payload.get('cmd'

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c4327cc16308bba8. Report an issue: GitHub.