What causes "NIXL transfer encountered ERR room={room}" in sgl-project/sglang?

Check the decode peer's health and logs — it likely crashed, OOMed, or restarted mid-transfer

sgl-project/sglang · error · RuntimeError

NIXL transfer encountered ERR room={room}

Error message

NIXL transfer encountered ERR room={room}

What it means

After posting transfers, transfer_worker polls agent.check_xfer_state(handle) for every handle; if any NIXL transfer handle reports state 'ERR', it raises RuntimeError naming the room. ERR means the underlying UCX/NIXL RDMA operation failed — e.g. remote memory deregistered, invalid descriptor, network/RDMA fault, or peer crash — so the request's KV transfer is marked Failed.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:1315

                        aux_xfer_handle = self.send_aux(
                            req.agent_name,
                            kv_chunk.prefill_aux_index,
                            dst_info.dst_aux_ptrs,
                            req.dst_aux_index,
                            aux_notif,
                        )
                        handles.append(aux_xfer_handle)

                if staging_deferred:
                    # Chunk has been re-enqueued; do not advance status.
                    continue

                while handles:
                    all_done = True
                    for handle in handles:
                        state = self.agent.check_xfer_state(handle)
                        if state == "ERR":
                            raise RuntimeError(
                                f"NIXL transfer encountered ERR room={room}"
                            )
                        if state != "DONE":
                            all_done = False
                    if all_done:
                        break
                    time.sleep(0)

                self._staging_outstanding[room] -= 1
                if self.enable_deferred_decode_kv_release:
                    # Handles all DONE => this room's writes landed; ack if it
                    # was aborted and nothing else is outstanding.
                    self._maybe_ack_drained_abort(room)
                if kv_chunk.is_last_chunk:
                    self.update_status(room, KVPoll.Success)
                elif self.check_status(room) != KVPoll.Success:
                    # A deferred earlier chunk can complete after the last chunk
                    # already concluded Success; don't regress the status.

View on GitHub (pinned to 0132848349)

Solutions

Check the decode peer's health and logs — it likely crashed, OOMed, or restarted mid-transfer
Verify the RDMA/UCX environment (UCX_NET_DEVICES, UCX_TLS, ibstat) and GPU-NIC affinity on both nodes
Ensure both PD nodes run the same SGLang/CUDA/UCX versions so registration geometry matches
Retry the request (the room is marked Failed; client retry re-runs prefill); if persistent, capture UCX logs (UCX_LOG_LEVEL=info) to isolate the transport error

Defensive patterns

Strategy: retry

Try / catch

except RuntimeError as e:
    if 'ERR room=' in str(e):
        log_room_state(room); check_peer_liveness(agent_name)
        if transport_transient(): retry_request_with_backoff()
        else: page_oncall()

Prevention

Health-check decode peers before dispatching prefill traffic
Set NIXL/UCX timeouts and monitor per-room transfer failure rates
Keep the RDMA fabric stable: verify ibstat, UCX_NET_DEVICES, GPU-NIC affinity before launch
Make client retries idempotent so failed rooms re-prefill cleanly

When it happens

Trigger: Any send_kvcache/send_aux/maybe_send_extra handle enters ERR state during the DONE-polling loop: causes include the decode peer deregistering memory or exiting mid-transfer, stale registration geometry (mismatched descriptors), or an actual RDMA/IB or TCP transport error.

Common situations: Decode node crashing, OOMing, or restarting while prefill is mid-transfer; stale KV registrations after peer re-registration; flaky RDMA fabric, wrong UCX_NET_DEVICES or GPU-NIC affinity issues; mem kind/device id mismatch between registered descriptors.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3c61f7dfcadf9583. Report an issue: GitHub.