What causes "NIXL KVReceiver Exception" in sgl-project/sglang?

Search decode and prefill logs for a more specific KVTransferError carrying the real failure_reason on another rank.

sgl-project/sglang · critical · KVTransferError

NIXL KVReceiver Exception

Error message

NIXL KVReceiver Exception

What it means

Raised by NixlKVReceiver.failure_exception on the decode side when a KV receive for a bootstrap room failed. If no specific reason was recorded it raises the generic 'NIXL KVReceiver Exception' with is_from_another_rank=True, meaning the detailed failure was on the prefill sender or a peer rank.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:3053

                        ]
                    )
            except zmq.ZMQError:
                self.kv_mgr.record_failure(
                    self.bootstrap_room,
                    f"_register_kv_args to prefill {bootstrap_info.get('rank_ip')}:{bootstrap_info.get('rank_port')} failed",
                )
                self.conclude_state = KVPoll.Failed
                self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed)
                return False
        return True

    def failure_exception(self):
        with self.kv_mgr.failure_lock:
            failure_reason = self.kv_mgr.failure_records.pop(self.bootstrap_room, None)
        is_propagated = failure_reason is None
        if is_propagated:
            failure_reason = "NIXL KVReceiver Exception"
        raise KVTransferError(
            self.bootstrap_room, failure_reason, is_from_another_rank=is_propagated
        )


class NixlKVBootstrapServer(CommonKVBootstrapServer):
    pass

View on GitHub (pinned to 0132848349)

Solutions

Search decode and prefill logs for a more specific KVTransferError carrying the real failure_reason on another rank.
Verify both endpoints share the same bootstrap server URL and room id and that the prefill instance didn't restart mid-request.
Check RDMA/NIXL health (ucx_info, NIC state) between nodes.
Retry the workload; if a rank crash recurs, address its root cause (OOM, driver) first.

Defensive patterns

Strategy: try-catch

Try / catch

try:
    kv_receiver.wait_transfer()
except KVTransferError as e:
    if e.is_from_another_rank:
        check_prefill_node_logs(e.bootstrap_room)
    requeue_or_fail_request(e.bootstrap_room)

Prevention

Avoid restarting prefill instances while decode transfers are in flight.
Validate bootstrap server reachability from both nodes at startup.
Set transfer timeouts so stalls fail fast with a recorded reason.

When it happens

Trigger: Scheduler calls failure_exception() on a NixlKVReceiver after kv_mgr.failure_records has (or lacks) an entry for the bootstrap room — e.g. sender aborted, transfer timeout, or peer decode rank crashed.

Common situations: Decode node in PD disaggregation hits NIXL transfer errors: prefill instance restarted mid-transfer, RDMA fabric issues, bootstrap mismatch, or a peer rank crashed leaving no propagated reason.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b29ffc67323686c7. Report an issue: GitHub.