{"record":{"id":"15be076a5113435f","repo":"sgl-project/sglang","slug":"timed-out-waiting-for-ack-from-flexkv-layerwise-wo","errorCode":null,"errorMessage":"Timed out waiting for ACK from FlexKV layerwise worker","messagePattern":"Timed out waiting for ACK from FlexKV layerwise worker","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py","lineNumber":894,"sourceCode":"\n                # Phase 2: send 16-byte metadata + per-counter FDs + read ACK.\n                num_counters = self.layer_done_counter.num_counters\n                metadata = struct.pack(\n                    \"iiii\",\n                    self.rank_info.tp_rank_per_node,\n                    self.model_config.tp_size_per_node,\n                    self.rank_info.num_layers_per_pp_stage,\n                    num_counters,\n                )\n                sock.sendall(metadata)\n                for counter_id in range(num_counters):\n                    fds = self.layer_done_counter.events[counter_id].load_event_fds\n                    send_fds(sock, fds, struct.pack(\"i\", counter_id))\n                sock.settimeout(30.0)\n                try:\n                    ack = sock.recv(1)\n                except socket.timeout as exc:\n                    raise RuntimeError(\n                        \"Timed out waiting for ACK from FlexKV layerwise worker\"\n                    ) from exc\n                if not ack or ack[0] != 1:\n                    raise RuntimeError(\n                        f\"FlexKV layerwise worker NACK'd eventfd transfer \"\n                        f\"(ack={ack!r})\"\n                    )\n                logger.info(\n                    \"[FlexKV] Eventfd handshake complete %s counters=%d layers=%d\",\n                    self._label,\n                    num_counters,\n                    self.rank_info.num_layers_per_pp_stage,\n                )\n                return\n            except Exception as exc:  # noqa: BLE001\n                last_error = exc\n                logger.warning(\n                    \"[FlexKV] Eventfd handshake send_attempt=%d/%d failed: %s\",","sourceCodeStart":876,"sourceCodeEnd":912,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py#L876-L912","documentation":"After sending eventfds for a layer_done_counter to the worker over the socket, the connector waits 30s (sock.settimeout(30.0)) for a 1-byte ACK. A socket.timeout is converted to RuntimeError: the worker accepted the connection but never responded in time.","triggerScenarios":"The worker received fds but is blocked/hung (e.g. stuck in a CUDA or FlexKV call), its eventfd registration path deadlocked, or the machine is heavily loaded so the ACK is delayed past 30s.","commonSituations":"Worker hang due to a driver/FlexKV internal error; oversized batch of counters making registration slow; transient extreme load during startup; worker crashed after accept but before recv/send.","solutions":["Inspect the layerwise worker's stack (py-spy/gdb) to find where it stalled during eventfd registration","Check for worker crash logs / core dumps at the same timestamp; restart worker and connector together","If load-related, raise the 30s ACK timeout or reduce the number of counters registered per connection"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        connector = FlexKVConnector(...)\n        break\n    except RuntimeError as e:\n        if 'Timed out waiting for ACK' in str(e) and attempt < 2:\n            logger.warning('ACK timeout, restarting worker and retrying')\n            restart_flexkv_worker()\n            continue\n        raise","preventionTips":["Keep the worker's logs correlated by timestamp so ACK timeouts can be matched to worker stalls","Cap concurrent counter registrations during init to bound ACK latency below the 30s timeout"],"tags":["flexkv","timeout","eventfd","worker-hang"],"backgroundTag":"worker-ack-timeout","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}