{"record":{"id":"3c61f7dfcadf9583","repo":"sgl-project/sglang","slug":"nixl-transfer-encountered-err-room-room","errorCode":null,"errorMessage":"NIXL transfer encountered ERR room={room}","messagePattern":"NIXL transfer encountered ERR room=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":1315,"sourceCode":"                        aux_xfer_handle = self.send_aux(\n                            req.agent_name,\n                            kv_chunk.prefill_aux_index,\n                            dst_info.dst_aux_ptrs,\n                            req.dst_aux_index,\n                            aux_notif,\n                        )\n                        handles.append(aux_xfer_handle)\n\n                if staging_deferred:\n                    # Chunk has been re-enqueued; do not advance status.\n                    continue\n\n                while handles:\n                    all_done = True\n                    for handle in handles:\n                        state = self.agent.check_xfer_state(handle)\n                        if state == \"ERR\":\n                            raise RuntimeError(\n                                f\"NIXL transfer encountered ERR room={room}\"\n                            )\n                        if state != \"DONE\":\n                            all_done = False\n                    if all_done:\n                        break\n                    time.sleep(0)\n\n                self._staging_outstanding[room] -= 1\n                if self.enable_deferred_decode_kv_release:\n                    # Handles all DONE => this room's writes landed; ack if it\n                    # was aborted and nothing else is outstanding.\n                    self._maybe_ack_drained_abort(room)\n                if kv_chunk.is_last_chunk:\n                    self.update_status(room, KVPoll.Success)\n                elif self.check_status(room) != KVPoll.Success:\n                    # A deferred earlier chunk can complete after the last chunk\n                    # already concluded Success; don't regress the status.","sourceCodeStart":1297,"sourceCodeEnd":1333,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L1297-L1333","documentation":"After posting transfers, transfer_worker polls agent.check_xfer_state(handle) for every handle; if any NIXL transfer handle reports state 'ERR', it raises RuntimeError naming the room. ERR means the underlying UCX/NIXL RDMA operation failed — e.g. remote memory deregistered, invalid descriptor, network/RDMA fault, or peer crash — so the request's KV transfer is marked Failed.","triggerScenarios":"Any send_kvcache/send_aux/maybe_send_extra handle enters ERR state during the DONE-polling loop: causes include the decode peer deregistering memory or exiting mid-transfer, stale registration geometry (mismatched descriptors), or an actual RDMA/IB or TCP transport error.","commonSituations":"Decode node crashing, OOMing, or restarting while prefill is mid-transfer; stale KV registrations after peer re-registration; flaky RDMA fabric, wrong UCX_NET_DEVICES or GPU-NIC affinity issues; mem kind/device id mismatch between registered descriptors.","solutions":["Check the decode peer's health and logs — it likely crashed, OOMed, or restarted mid-transfer","Verify the RDMA/UCX environment (UCX_NET_DEVICES, UCX_TLS, ibstat) and GPU-NIC affinity on both nodes","Ensure both PD nodes run the same SGLang/CUDA/UCX versions so registration geometry matches","Retry the request (the room is marked Failed; client retry re-runs prefill); if persistent, capture UCX logs (UCX_LOG_LEVEL=info) to isolate the transport error"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"except RuntimeError as e:\n    if 'ERR room=' in str(e):\n        log_room_state(room); check_peer_liveness(agent_name)\n        if transport_transient(): retry_request_with_backoff()\n        else: page_oncall()","preventionTips":["Health-check decode peers before dispatching prefill traffic","Set NIXL/UCX timeouts and monitor per-room transfer failure rates","Keep the RDMA fabric stable: verify ibstat, UCX_NET_DEVICES, GPU-NIC affinity before launch","Make client retries idempotent so failed rooms re-prefill cleanly"],"tags":["nixl","rdma","network","disaggregation","transfer-failure"],"backgroundTag":"rdma-transfer-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}