{"record":{"id":"11940796970cc2ed","repo":"sgl-project/sglang","slug":"kvtransfererror-self-bootstrap-room-failure-reaso","errorCode":null,"errorMessage":"KVTransferError(self.bootstrap_room, failure_reason)","messagePattern":"KVTransferError\\(self\\.bootstrap_room, failure_reason\\)","errorType":"exception","errorClass":"KVTransferError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":2832,"sourceCode":"            }\n\n    def failure_exception(self):\n        exc = self.kv_mgr.exceptions.pop(self.bootstrap_room, None)\n        with self.kv_mgr.failure_lock:\n            failure_reason = self.kv_mgr.failure_records.pop(self.bootstrap_room, None)\n\n        if self.conclude_state is None:\n            self.conclude_state = KVPoll.Failed\n        self._send_failed = True\n\n        self.clear()\n\n        if self._send_error is not None:\n            raise self._send_error\n        if exc is not None:\n            raise exc\n        if failure_reason is not None:\n            raise KVTransferError(self.bootstrap_room, failure_reason)\n        raise KVTransferError(\n            self.bootstrap_room, \"NIXL KVSender Exception\", is_from_another_rank=True\n        )\n\n\nclass NixlKVReceiver(CommonKVReceiver):\n    def __init__(\n        self,\n        mgr: NixlKVManager,\n        bootstrap_addr: str,\n        bootstrap_room: Optional[int] = None,\n    ):\n        self.started_transfer = False\n        super().__init__(mgr, bootstrap_addr, bootstrap_room)\n        self.init_time = None\n\n    def send_metadata(\n        self,","sourceCodeStart":2814,"sourceCodeEnd":2850,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L2814-L2850","documentation":"Raised by NixlKVSender.failure_exception when the sender for a bootstrap room has failed and a concrete failure reason was recorded by a peer rank or the KV manager. It wraps KVTransferError with the bootstrap room id and the propagated failure reason, signaling the prefill-side KV transfer over NIXL aborted.","triggerScenarios":"Calling failure_exception() on a NixlKVSender after a transfer error was recorded in kv_mgr.failure_records under this bootstrap_room (e.g. remote side closed, NIXL agent error, or an earlier per-send exception on another rank).","commonSituations":"Disaggregated prefill/decode (PD disaggregation) with the NIXL backend: the decode receiver disconnects, UCX/NIXL fabric errors, or one TP rank fails and the reason is broadcast so all ranks raise consistently.","solutions":["Check scheduler/NIXL logs for the underlying failure_reason string; it names the true root cause (often a peer rank's exception).","Verify peer connectivity: bootstrap server reachable, same --disaggregation-bootstrap-room scope, NICs/UCX_TLS settings consistent across prefill and decode nodes.","Restart the failed prefill/decode instance; PD transfer failures are usually fatal to the affected requests and require re-dispatch.","If persistent, switch transfer backend (e.g. Mooncake) or update NIXL/UCX versions to match between endpoints."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"from sglang.srt.disaggregation.base import KVTransferError\ntry:\n    kv_sender.check_complete()  # or await transfer completion\nexcept KVTransferError as e:\n    logger.error(f\"room={e.bootstrap_room} reason={e.message}\")\n    scheduler.abort_request(req, abort_reason=e.message)","preventionTips":["Monitor both prefill and decode node health so peer failures surface early.","Keep NIXL/UCX versions identical across nodes.","Log bootstrap_room ids to correlate failures across ranks."],"tags":["sglang","nixl","kv-transfer","disaggregation","distributed-inference"],"backgroundTag":"distributed-kv-transfer-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}