sgl-project/sglang · critical · KVTransferError

NIXL KVSender Exception

Error message

NIXL KVSender Exception

What it means

Fallback error from NixlKVSender.failure_exception: no prior _send_error, no propagated exception, and no recorded failure reason, so the sender cannot identify which rank failed. The flag is_from_another_rank=True indicates the true failure occurred on a different rank of the TP group.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:2833

    def failure_exception(self):
        exc = self.kv_mgr.exceptions.pop(self.bootstrap_room, None)
        with self.kv_mgr.failure_lock:
            failure_reason = self.kv_mgr.failure_records.pop(self.bootstrap_room, None)

        if self.conclude_state is None:
            self.conclude_state = KVPoll.Failed
        self._send_failed = True

        self.clear()

        if self._send_error is not None:
            raise self._send_error
        if exc is not None:
            raise exc
        if failure_reason is not None:
            raise KVTransferError(self.bootstrap_room, failure_reason)
        raise KVTransferError(
            self.bootstrap_room, "NIXL KVSender Exception", is_from_another_rank=True
        )


class NixlKVReceiver(CommonKVReceiver):
    def __init__(
        self,
        mgr: NixlKVManager,
        bootstrap_addr: str,
        bootstrap_room: Optional[int] = None,
    ):
        self.started_transfer = False
        super().__init__(mgr, bootstrap_addr, bootstrap_room)
        self.init_time = None

    def send_metadata(
        self,
        kv_indices: npt.NDArray[np.int32],

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect logs/process state of ALL TP ranks (especially dmesg / OOM killer) — the informative traceback is on another rank.
2. Check for GPU OOM or NCCL/UCX timeouts on peer ranks and fix the root cause there.
3. Restart the prefill worker group so the ranks re-bootstrap.
4. If root cause is recurring OOM, lower --max-total-tokens / mem-fraction or chunked_prefill_size.

try:
    await sender_transfer_done
except KVTransferError as e:
    if e.is_from_another_rank:
        # inspect peer rank logs; no local state can explain it
        escalate_to_rank_diagnostics(e.bootstrap_room)
    abort_request(e.bootstrap_room)

When it happens

Trigger: failure_exception() is invoked (typically by the scheduler aborting the transfer) when a sibling TP rank died without this rank receiving a specific error or failure record for the bootstrap room.

Common situations: One GPU/TP rank in a disaggregated prefill deployment OOMs or crashes; surviving ranks detect the transfer stall and raise this generic propagated error because the detailed reason lives only on the crashed rank.

Related errors

• KVTransferError(self.bootstrap_room, failure_reason)
• NIXL KVReceiver Exception
• SGLANG_DISAGG_STAGING_BUFFER with pp_size > 1 is only suppor
• MiniMaxH3Pipeline only supports monolithic deployment; disag
• Pi05Pipeline v1 supports same-process execution only. Use pr

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/93888c10040b6c24. Report an issue: GitHub.