vllm-project/vllm · critical · BenchError

IO error: {0}

Error message

IO error: {0}

What it means

While waiting for outstanding NIXL expert-weight transfers, _wait_for_all_transfers polls check_xfer_state. A handle returned a state other than DONE or PROC (e.g. ERR), meaning the underlying one-sided transfer failed on the fabric.

Source

Thrown at rust/src/bench/src/error.rs:33

    Tokenizer(String),

    /// The server's /tokenize//detokenize endpoint is not usable (4xx status:
    /// not exposed, or rejected by a gateway such as LLM-d/EPP that returns
    /// 400 instead of 404). Callers treat this as "skip verification", unlike
    /// `Tokenizer` errors which are genuine failures.
    #[error("tokenize endpoint unavailable: {0}")]
    TokenizeUnavailable(String),

    #[error("Configuration error: {0}")]
    Config(String),

    #[error("Endpoint not ready after {0}s: {1}")]
    EndpointTimeout(u64, String),

    #[error("Backend error: {0}")]
    Backend(String),

    #[error("IO error: {0}")]
    Io(#[from] std::io::Error),
}

pub type Result<T> = std::result::Result<T, BenchError>;

View on GitHub (pinned to c794754062)

Solutions

1. Check whether the peer rank crashed (look at other ranks' logs / process liveness) — a dead peer is the most common cause
2. Inspect UCX logs (UCX_LOG_LEVEL=info) and NIC/fabric errors on the affected nodes
3. Retry the EPLB rebalance after restoring all ranks to a healthy state; NIXL transfers are not resumable mid-flight

try:
    comm.rebalance(...)  # drives _wait_for_all_transfers
except RuntimeError as e:
    if 'NIXL transfer failed' in str(e):
        if peer_rank_alive():
            rebuild_transfers_and_retry()  # re-run EPLB step from a consistent state
        else:
            evict_peer_and_rescale()  # peer died: restart EPLB without it
    else:
        raise

When it happens

Trigger: A NIXL GTO descriptor transfer failing during EPLB rebalancing: remote agent crash mid-transfer, RMA error from UCX, or an invalid/unregistered remote descriptor being written.

Common situations: A peer rank dying (OOM, preemption) during EPLB weight redistribution; transient InfiniBand/RDMA errors; UCX misconfiguration under load.

Related errors

• Configuration error: {0}
• Endpoint not ready after {0}s: {1}
• Backend error: {0}
• multimodal input is not supported by this chat renderer
• unsupported multimodal content: {0}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/7efc4e311fae74b3. Report an issue: GitHub.