What causes "KVTransferError(self.bootstrap_room, failure_reason)" in sgl-project/sglang?

Check scheduler/NIXL logs for the underlying failure_reason string; it names the true root cause (often a peer rank's exception).

sgl-project/sglang · critical · KVTransferError

KVTransferError(self.bootstrap_room, failure_reason)

Error message

KVTransferError(self.bootstrap_room, failure_reason)

What it means

Raised by NixlKVSender.failure_exception when the sender for a bootstrap room has failed and a concrete failure reason was recorded by a peer rank or the KV manager. It wraps KVTransferError with the bootstrap room id and the propagated failure reason, signaling the prefill-side KV transfer over NIXL aborted.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:2832

            }

    def failure_exception(self):
        exc = self.kv_mgr.exceptions.pop(self.bootstrap_room, None)
        with self.kv_mgr.failure_lock:
            failure_reason = self.kv_mgr.failure_records.pop(self.bootstrap_room, None)

        if self.conclude_state is None:
            self.conclude_state = KVPoll.Failed
        self._send_failed = True

        self.clear()

        if self._send_error is not None:
            raise self._send_error
        if exc is not None:
            raise exc
        if failure_reason is not None:
            raise KVTransferError(self.bootstrap_room, failure_reason)
        raise KVTransferError(
            self.bootstrap_room, "NIXL KVSender Exception", is_from_another_rank=True
        )


class NixlKVReceiver(CommonKVReceiver):
    def __init__(
        self,
        mgr: NixlKVManager,
        bootstrap_addr: str,
        bootstrap_room: Optional[int] = None,
    ):
        self.started_transfer = False
        super().__init__(mgr, bootstrap_addr, bootstrap_room)
        self.init_time = None

    def send_metadata(
        self,

View on GitHub (pinned to 0132848349)

Solutions

Check scheduler/NIXL logs for the underlying failure_reason string; it names the true root cause (often a peer rank's exception).
Verify peer connectivity: bootstrap server reachable, same --disaggregation-bootstrap-room scope, NICs/UCX_TLS settings consistent across prefill and decode nodes.
Restart the failed prefill/decode instance; PD transfer failures are usually fatal to the affected requests and require re-dispatch.
If persistent, switch transfer backend (e.g. Mooncake) or update NIXL/UCX versions to match between endpoints.

Defensive patterns

Strategy: try-catch

Try / catch

from sglang.srt.disaggregation.base import KVTransferError
try:
    kv_sender.check_complete()  # or await transfer completion
except KVTransferError as e:
    logger.error(f"room={e.bootstrap_room} reason={e.message}")
    scheduler.abort_request(req, abort_reason=e.message)

Prevention

Monitor both prefill and decode node health so peer failures surface early.
Keep NIXL/UCX versions identical across nodes.
Log bootstrap_room ids to correlate failures across ranks.

When it happens

Trigger: Calling failure_exception() on a NixlKVSender after a transfer error was recorded in kv_mgr.failure_records under this bootstrap_room (e.g. remote side closed, NIXL agent error, or an earlier per-send exception on another rank).

Common situations: Disaggregated prefill/decode (PD disaggregation) with the NIXL backend: the decode receiver disconnects, UCX/NIXL fabric errors, or one TP rank fails and the reason is broadcast so all ranks raise consistently.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/11940796970cc2ed. Report an issue: GitHub.