{"record":{"id":"d293f167bbc42a5c","repo":"sgl-project/sglang","slug":"flexkv-layerwise-worker-nack-d-eventfd-transfer-a","errorCode":null,"errorMessage":"FlexKV layerwise worker NACK'd eventfd transfer (ack={ack!r})","messagePattern":"FlexKV layerwise worker NACK'd eventfd transfer \\(ack=(.+?)\\)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py","lineNumber":898,"sourceCode":"                    \"iiii\",\n                    self.rank_info.tp_rank_per_node,\n                    self.model_config.tp_size_per_node,\n                    self.rank_info.num_layers_per_pp_stage,\n                    num_counters,\n                )\n                sock.sendall(metadata)\n                for counter_id in range(num_counters):\n                    fds = self.layer_done_counter.events[counter_id].load_event_fds\n                    send_fds(sock, fds, struct.pack(\"i\", counter_id))\n                sock.settimeout(30.0)\n                try:\n                    ack = sock.recv(1)\n                except socket.timeout as exc:\n                    raise RuntimeError(\n                        \"Timed out waiting for ACK from FlexKV layerwise worker\"\n                    ) from exc\n                if not ack or ack[0] != 1:\n                    raise RuntimeError(\n                        f\"FlexKV layerwise worker NACK'd eventfd transfer \"\n                        f\"(ack={ack!r})\"\n                    )\n                logger.info(\n                    \"[FlexKV] Eventfd handshake complete %s counters=%d layers=%d\",\n                    self._label,\n                    num_counters,\n                    self.rank_info.num_layers_per_pp_stage,\n                )\n                return\n            except Exception as exc:  # noqa: BLE001\n                last_error = exc\n                logger.warning(\n                    \"[FlexKV] Eventfd handshake send_attempt=%d/%d failed: %s\",\n                    send_attempt + 1,\n                    max_send_retries,\n                    exc,\n                )","sourceCodeStart":880,"sourceCodeEnd":916,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py#L880-L916","documentation":"The worker replies with a 1-byte ACK: 1 for success. An empty recv (worker closed) or any value other than 1 is treated as a NACK and raised as RuntimeError — the worker explicitly rejected the eventfd transfer, likely failing to import the fds or register the counter.","triggerScenarios":"Worker-side import of the SCM_RIGHTS fds failed (fd table full, wrong counter_id, duplicate counter registration), so it replies with a non-1 status or closes the socket; also protocol mismatch on the ACK encoding between versions.","commonSituations":"Registering the same counter_id twice after a partial retry; worker and scheduler built from different versions with different ACK semantics; file-descriptor exhaustion on the worker.","solutions":["Check worker logs immediately around the NACK — it logs the specific registration failure","Ensure counter ids are unique per fkv_task_id and never re-registered after NACK without cleanup","Align scheduler and worker versions; verify ulimit -n on the worker host"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    connector = FlexKVConnector(...)\nexcept RuntimeError as e:\n    if \"NACK'd eventfd transfer\" in str(e):\n        logger.error('worker rejected eventfds: %s; disabling layerwise offload', e)\n        connector = FlexKVConnector(..., layerwise_enabled=False)\n    else:\n        raise","preventionTips":["Ensure counter ids are registered exactly once per fkv_task_id","Check ulimit -n on worker hosts to avoid fd exhaustion during NACK storms"],"tags":["flexkv","eventfd","nack","file-descriptors"],"backgroundTag":"worker-ack-negative","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}