{"record":{"id":"b29ffc67323686c7","repo":"sgl-project/sglang","slug":"nixl-kvreceiver-exception","errorCode":null,"errorMessage":"NIXL KVReceiver Exception","messagePattern":"NIXL KVReceiver Exception","errorType":"exception","errorClass":"KVTransferError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":3053,"sourceCode":"                        ]\n                    )\n            except zmq.ZMQError:\n                self.kv_mgr.record_failure(\n                    self.bootstrap_room,\n                    f\"_register_kv_args to prefill {bootstrap_info.get('rank_ip')}:{bootstrap_info.get('rank_port')} failed\",\n                )\n                self.conclude_state = KVPoll.Failed\n                self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed)\n                return False\n        return True\n\n    def failure_exception(self):\n        with self.kv_mgr.failure_lock:\n            failure_reason = self.kv_mgr.failure_records.pop(self.bootstrap_room, None)\n        is_propagated = failure_reason is None\n        if is_propagated:\n            failure_reason = \"NIXL KVReceiver Exception\"\n        raise KVTransferError(\n            self.bootstrap_room, failure_reason, is_from_another_rank=is_propagated\n        )\n\n\nclass NixlKVBootstrapServer(CommonKVBootstrapServer):\n    pass\n","sourceCodeStart":3035,"sourceCodeEnd":3060,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L3035-L3060","documentation":"Raised by NixlKVReceiver.failure_exception on the decode side when a KV receive for a bootstrap room failed. If no specific reason was recorded it raises the generic 'NIXL KVReceiver Exception' with is_from_another_rank=True, meaning the detailed failure was on the prefill sender or a peer rank.","triggerScenarios":"Scheduler calls failure_exception() on a NixlKVReceiver after kv_mgr.failure_records has (or lacks) an entry for the bootstrap room — e.g. sender aborted, transfer timeout, or peer decode rank crashed.","commonSituations":"Decode node in PD disaggregation hits NIXL transfer errors: prefill instance restarted mid-transfer, RDMA fabric issues, bootstrap mismatch, or a peer rank crashed leaving no propagated reason.","solutions":["Search decode and prefill logs for a more specific KVTransferError carrying the real failure_reason on another rank.","Verify both endpoints share the same bootstrap server URL and room id and that the prefill instance didn't restart mid-request.","Check RDMA/NIXL health (ucx_info, NIC state) between nodes.","Retry the workload; if a rank crash recurs, address its root cause (OOM, driver) first."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    kv_receiver.wait_transfer()\nexcept KVTransferError as e:\n    if e.is_from_another_rank:\n        check_prefill_node_logs(e.bootstrap_room)\n    requeue_or_fail_request(e.bootstrap_room)","preventionTips":["Avoid restarting prefill instances while decode transfers are in flight.","Validate bootstrap server reachability from both nodes at startup.","Set transfer timeouts so stalls fail fast with a recorded reason."],"tags":["sglang","nixl","kv-receive","disaggregation","distributed-inference"],"backgroundTag":"distributed-kv-transfer-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}