sgl-project/sglang · critical · RuntimeError

FlexKV layerwise worker NACK'd eventfd transfer (ack={ack!r}

Error message

FlexKV layerwise worker NACK'd eventfd transfer (ack={ack!r})

What it means

The worker replies with a 1-byte ACK: 1 for success. An empty recv (worker closed) or any value other than 1 is treated as a NACK and raised as RuntimeError — the worker explicitly rejected the eventfd transfer, likely failing to import the fds or register the counter.

Source

Thrown at python/sglang/srt/mem_cache/storage/flexkv/flexkv_connector.py:898

                    "iiii",
                    self.rank_info.tp_rank_per_node,
                    self.model_config.tp_size_per_node,
                    self.rank_info.num_layers_per_pp_stage,
                    num_counters,
                )
                sock.sendall(metadata)
                for counter_id in range(num_counters):
                    fds = self.layer_done_counter.events[counter_id].load_event_fds
                    send_fds(sock, fds, struct.pack("i", counter_id))
                sock.settimeout(30.0)
                try:
                    ack = sock.recv(1)
                except socket.timeout as exc:
                    raise RuntimeError(
                        "Timed out waiting for ACK from FlexKV layerwise worker"
                    ) from exc
                if not ack or ack[0] != 1:
                    raise RuntimeError(
                        f"FlexKV layerwise worker NACK'd eventfd transfer "
                        f"(ack={ack!r})"
                    )
                logger.info(
                    "[FlexKV] Eventfd handshake complete %s counters=%d layers=%d",
                    self._label,
                    num_counters,
                    self.rank_info.num_layers_per_pp_stage,
                )
                return
            except Exception as exc:  # noqa: BLE001
                last_error = exc
                logger.warning(
                    "[FlexKV] Eventfd handshake send_attempt=%d/%d failed: %s",
                    send_attempt + 1,
                    max_send_retries,
                    exc,
                )

View on GitHub (pinned to 0132848349)

Solutions

1. Check worker logs immediately around the NACK — it logs the specific registration failure
2. Ensure counter ids are unique per fkv_task_id and never re-registered after NACK without cleanup
3. Align scheduler and worker versions; verify ulimit -n on the worker host

try:
    connector = FlexKVConnector(...)
except RuntimeError as e:
    if "NACK'd eventfd transfer" in str(e):
        logger.error('worker rejected eventfds: %s; disabling layerwise offload', e)
        connector = FlexKVConnector(..., layerwise_enabled=False)
    else:
        raise

When it happens

Trigger: Worker-side import of the SCM_RIGHTS fds failed (fd table full, wrong counter_id, duplicate counter registration), so it replies with a non-1 status or closes the socket; also protocol mismatch on the ACK encoding between versions.

Common situations: Registering the same counter_id twice after a partial retry; worker and scheduler built from different versions with different ACK semantics; file-descriptor exhaustion on the worker.

Related errors

• Timed out waiting for ACK from FlexKV layerwise worker
• [FlexKV] Failed to send eventfds to {self._layerwise_socket}
• Unsupported KV cache type {type(kvcache).__name__}: expected
• Tag mismatch: expected CMD_LAYERWISE, got {payload.get('cmd'
• store_kv: token_ids has {n} entries but kv_indices has {len(

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d293f167bbc42a5c. Report an issue: GitHub.