sgl-project/sglang · critical · KVTransferError

KVTransferError

Error message

KVTransferError

What it means

failure_exception raises KVTransferError carrying the bootstrap_room and the recorded failure reason; if this rank has no record the reason is propagated from another rank. It converts silent KV-transfer corruption into an explicit per-room failure.

Source

Thrown at python/sglang/srt/disaggregation/mooncake/conn.py:2256

    state, then raise with the recorded reason -- or, when no reason was
    recorded locally, report it as propagated from another rank. Expects the
    concrete class to provide ``conclude_state``, ``clear()``,
    ``bootstrap_room`` and ``kv_mgr``.
    """

    def failure_exception(self):
        # A room with no locally recorded reason failed on another rank.
        if self.conclude_state is None:
            self.conclude_state = KVPoll.Failed

        self.clear()

        with self.kv_mgr.failure_lock:
            failure_reason = self.kv_mgr.failure_records.pop(self.bootstrap_room, None)
        is_propagated = failure_reason is None
        if is_propagated:
            failure_reason = "Failed due to an unknown reason from another rank"
        raise KVTransferError(
            self.bootstrap_room, failure_reason, is_from_another_rank=is_propagated
        )


class MooncakeKVSender(MooncakeFailureExceptionMixin, CommonKVSender):

    def __init__(
        self,
        mgr: MooncakeKVManager,
        bootstrap_addr: str,
        bootstrap_room: int,
        dest_tp_ranks: List[int],
        pp_rank: int,
        req_has_disagg_prefill_dp_rank: bool = False,
    ):
        super().__init__(
            mgr,
            bootstrap_addr,

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect KVTransferError.failure_reason and the originating rank's logs for the true failure
2. Fix the root cause (fabric, registration, layout) — this error is the reporting channel, not the bug
3. Retry the request once the PD pair is healthy

def is_kv_transfer_error(e: BaseException) -> bool:
    return type(e).__name__ == "KVTransferError"

try: ... except KVTransferError as e: log(e.failure_reason, e.is_from_another_rank); fail/retry the request's room, not the server

When it happens

Trigger: A transfer failure recorded against the bootstrap room (any rank failure) and later surfaced via failure_exception on the sender/receiver.

Common situations: One rank of a multi-TP PD transfer failing, causing all sibling rooms to fail with propagated reasons; surfaced during polling of transfer status.

Related errors

• Error on rank 0
• Error on rank 0: {broadcasted_batch}
• MiniMax H3 text encode failed on rank {owner}: {owner_error}
• Page size mismatch: prefill server has page_size={info.page_
• KV cache dtype mismatch: prefill server has kv_cache_dtype={

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/44e9b099c17fc6a5. Report an issue: GitHub.