{"record":{"id":"93888c10040b6c24","repo":"sgl-project/sglang","slug":"nixl-kvsender-exception","errorCode":null,"errorMessage":"NIXL KVSender Exception","messagePattern":"NIXL KVSender Exception","errorType":"exception","errorClass":"KVTransferError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":2833,"sourceCode":"\n    def failure_exception(self):\n        exc = self.kv_mgr.exceptions.pop(self.bootstrap_room, None)\n        with self.kv_mgr.failure_lock:\n            failure_reason = self.kv_mgr.failure_records.pop(self.bootstrap_room, None)\n\n        if self.conclude_state is None:\n            self.conclude_state = KVPoll.Failed\n        self._send_failed = True\n\n        self.clear()\n\n        if self._send_error is not None:\n            raise self._send_error\n        if exc is not None:\n            raise exc\n        if failure_reason is not None:\n            raise KVTransferError(self.bootstrap_room, failure_reason)\n        raise KVTransferError(\n            self.bootstrap_room, \"NIXL KVSender Exception\", is_from_another_rank=True\n        )\n\n\nclass NixlKVReceiver(CommonKVReceiver):\n    def __init__(\n        self,\n        mgr: NixlKVManager,\n        bootstrap_addr: str,\n        bootstrap_room: Optional[int] = None,\n    ):\n        self.started_transfer = False\n        super().__init__(mgr, bootstrap_addr, bootstrap_room)\n        self.init_time = None\n\n    def send_metadata(\n        self,\n        kv_indices: npt.NDArray[np.int32],","sourceCodeStart":2815,"sourceCodeEnd":2851,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L2815-L2851","documentation":"Fallback error from NixlKVSender.failure_exception: no prior _send_error, no propagated exception, and no recorded failure reason, so the sender cannot identify which rank failed. The flag is_from_another_rank=True indicates the true failure occurred on a different rank of the TP group.","triggerScenarios":"failure_exception() is invoked (typically by the scheduler aborting the transfer) when a sibling TP rank died without this rank receiving a specific error or failure record for the bootstrap room.","commonSituations":"One GPU/TP rank in a disaggregated prefill deployment OOMs or crashes; surviving ranks detect the transfer stall and raise this generic propagated error because the detailed reason lives only on the crashed rank.","solutions":["Inspect logs/process state of ALL TP ranks (especially dmesg / OOM killer) — the informative traceback is on another rank.","Check for GPU OOM or NCCL/UCX timeouts on peer ranks and fix the root cause there.","Restart the prefill worker group so the ranks re-bootstrap.","If root cause is recurring OOM, lower --max-total-tokens / mem-fraction or chunked_prefill_size."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    await sender_transfer_done\nexcept KVTransferError as e:\n    if e.is_from_another_rank:\n        # inspect peer rank logs; no local state can explain it\n        escalate_to_rank_diagnostics(e.bootstrap_room)\n    abort_request(e.bootstrap_room)","preventionTips":["Set GPU memory limits to avoid peer-rank OOM.","Enable per-rank logging/watchdogs on all TP ranks.","Treat generic propagated errors as peer-crash indicators and check dmesg."],"tags":["sglang","nixl","kv-transfer","disaggregation","tp-rank-crash"],"backgroundTag":"distributed-kv-transfer-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}