{"record":{"id":"7efc4e311fae74b3","repo":"vllm-project/vllm","slug":"io-error-0","errorCode":null,"errorMessage":"IO error: {0}","messagePattern":"IO error: (.+?)","errorType":"exception","errorClass":"BenchError","httpStatus":null,"severity":"critical","filePath":"rust/src/bench/src/error.rs","lineNumber":33,"sourceCode":"    Tokenizer(String),\n\n    /// The server's /tokenize//detokenize endpoint is not usable (4xx status:\n    /// not exposed, or rejected by a gateway such as LLM-d/EPP that returns\n    /// 400 instead of 404). Callers treat this as \"skip verification\", unlike\n    /// `Tokenizer` errors which are genuine failures.\n    #[error(\"tokenize endpoint unavailable: {0}\")]\n    TokenizeUnavailable(String),\n\n    #[error(\"Configuration error: {0}\")]\n    Config(String),\n\n    #[error(\"Endpoint not ready after {0}s: {1}\")]\n    EndpointTimeout(u64, String),\n\n    #[error(\"Backend error: {0}\")]\n    Backend(String),\n\n    #[error(\"IO error: {0}\")]\n    Io(#[from] std::io::Error),\n}\n\npub type Result<T> = std::result::Result<T, BenchError>;\n","sourceCodeStart":15,"sourceCodeEnd":38,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/bench/src/error.rs#L15-L38","documentation":"While waiting for outstanding NIXL expert-weight transfers, _wait_for_all_transfers polls check_xfer_state. A handle returned a state other than DONE or PROC (e.g. ERR), meaning the underlying one-sided transfer failed on the fabric.","triggerScenarios":"A NIXL GTO descriptor transfer failing during EPLB rebalancing: remote agent crash mid-transfer, RMA error from UCX, or an invalid/unregistered remote descriptor being written.","commonSituations":"A peer rank dying (OOM, preemption) during EPLB weight redistribution; transient InfiniBand/RDMA errors; UCX misconfiguration under load.","solutions":["Check whether the peer rank crashed (look at other ranks' logs / process liveness) — a dead peer is the most common cause","Inspect UCX logs (UCX_LOG_LEVEL=info) and NIC/fabric errors on the affected nodes","Retry the EPLB rebalance after restoring all ranks to a healthy state; NIXL transfers are not resumable mid-flight"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    comm.rebalance(...)  # drives _wait_for_all_transfers\nexcept RuntimeError as e:\n    if 'NIXL transfer failed' in str(e):\n        if peer_rank_alive():\n            rebuild_transfers_and_retry()  # re-run EPLB step from a consistent state\n        else:\n            evict_peer_and_rescale()  # peer died: restart EPLB without it\n    else:\n        raise","preventionTips":["Monitor peer liveness during EPLB rebalance windows","Set UCX log level and fabric health checks before large weight redistributions","Treat this error as terminal for the current rebalance — restart from consistent state rather than resuming"],"tags":["nixl","eplb","rdma","transfer-failure","vllm"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}