{"record":{"id":"c4327cc16308bba8","repo":"sgl-project/sglang","slug":"flexkv-failed-to-send-eventfds-to-self-layerwi","errorCode":null,"errorMessage":"[FlexKV] Failed to send eventfds to {self._layerwise_socket} after {max_send_retries} attempts: {last_error}","messagePattern":"\\[FlexKV\\] Failed to send eventfds to (.+?) after (.+?) attempts: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py","lineNumber":922,"sourceCode":"                    self._label,\n                    num_counters,\n                    self.rank_info.num_layers_per_pp_stage,\n                )\n                return\n            except Exception as exc:  # noqa: BLE001\n                last_error = exc\n                logger.warning(\n                    \"[FlexKV] Eventfd handshake send_attempt=%d/%d failed: %s\",\n                    send_attempt + 1,\n                    max_send_retries,\n                    exc,\n                )\n            finally:\n                if sock is not None:\n                    sock.close()\n                time.sleep(retry_interval)\n\n        raise RuntimeError(\n            f\"[FlexKV] Failed to send eventfds to {self._layerwise_socket} \"\n            f\"after {max_send_retries} attempts: {last_error}\"\n        )\n","sourceCodeStart":904,"sourceCodeEnd":926,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py#L904-L926","documentation":"_send_eventfds_to_worker wraps the whole send phase in max_send_retries; any per-attempt exception (raise ... in the loop body) is retried after closing the socket. If every attempt fails, the last error is re-raised with this aggregated RuntimeError naming the socket path and retry count.","triggerScenarios":"Persistent send-phase failures: send_fds raising (broken pipe because the worker died mid-handshake), counter_id missing from layer_done_counter.events, or struct/serialization errors — repeated across all max_send_retries attempts.","commonSituations":"Worker crashing between connect and send; referencing an unregistered counter_id; repeated transient socket errors masking an underlying worker bug.","solutions":["Read last_error context in the log: broken pipe points to a dead worker, KeyError on events[...] to an unregistered counter","Reproduce with a single counter to isolate whether the failure is per-counter or per-connection","Fix the root cause on the worker (crash/registration), then restart both sides; increasing retries alone rarely helps here"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    _send_eventfds_to_worker_wrapper(...)\nexcept RuntimeError as e:\n    if 'Failed to send eventfds' in str(e):\n        capture_last_error_from_log(e)  # inspect nested cause for broken pipe vs KeyError\n        restart_flexkv_worker()\n        raise","preventionTips":["Pre-register all counter ids in layer_done_counter.events before opening the socket","Treat exhausted send retries as a worker-liveness incident, not a transient network issue"],"tags":["flexkv","eventfd","retry-exhausted","unix-socket"],"backgroundTag":"socket-send-exhausted-retries","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}