vllm-project/vllm · critical · BenchError

Tokenizer error: {0}

Error message

Tokenizer error: {0}

What it means

Raised by MoRIIOWrapper.waiting_for_transfer_complete (moriio_engine.py:696) after the wait loop finishes: it polls every RDMA transfer status (either the explicit transfer_statuses argument or all statuses drained from self.transfer_status) and collects two kinds of failures — terminal status.Failed() errors, and transfers still incomplete when the deadline (self._transfer_timeout, configurable via kv_connector_extra_config.transfer_timeout) expires. It then raises TransferError summarizing how many of the waited transfers failed and why. This is the single point where MoRIIO RDMA data-plane failures surface to the vLLM scheduler.

Source

Thrown at rust/src/bench/src/error.rs:14

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project

use thiserror::Error;

#[derive(Error, Debug)]
pub enum BenchError {
    #[error("HTTP request failed: {0}")]
    Http(#[from] reqwest::Error),

    #[error("JSON error: {0}")]
    Json(#[from] serde_json::Error),

    #[error("Tokenizer error: {0}")]
    Tokenizer(String),

    /// The server's /tokenize//detokenize endpoint is not usable (4xx status:
    /// not exposed, or rejected by a gateway such as LLM-d/EPP that returns
    /// 400 instead of 404). Callers treat this as "skip verification", unlike
    /// `Tokenizer` errors which are genuine failures.
    #[error("tokenize endpoint unavailable: {0}")]
    TokenizeUnavailable(String),

    #[error("Configuration error: {0}")]
    Config(String),

    #[error("Endpoint not ready after {0}s: {1}")]
    EndpointTimeout(u64, String),

    #[error("Backend error: {0}")]
    Backend(String),

View on GitHub (pinned to c794754062)

Solutions

1. Read the per-transfer lines in the message: 'RDMA transfer failed: ... (code=...)' points at protocol/MR issues, while 'timed out after Ns' points at capacity/latency.
2. For timeouts, raise the budget via kv_connector_extra_config.transfer_timeout, and/or reduce transfer concurrency (fewer requests in flight, smaller batched merges) as the error text itself suggests.
3. For 'SQ full' style failures, reduce the number of concurrent batch_read/batch_write calls per step or batch blocks into fewer larger transfers via merge_contiguous_blocks.
4. Verify both endpoints are alive and healthy: a crashed/restarted peer produces indefinite timeouts; check decode/prefill engine logs and re-establish the session.
5. Check RDMA fabric health (ibstat, ibping, link errors) and that memory registrations on both sides are current (re-register after any KV cache resize).

Example fix

// before
wrapper.waiting_for_transfer_complete(statuses)  # TransferError kills the step on any failure

// after (retry once on transient SQ-full, then surface)
from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_engine import TransferError

for attempt in range(2):
    failed = [s for s in statuses if s.Failed()]
    if not failed:
        break
    statuses = wrapper.retry_failed_reads(failed)  # re-post reads that failed transiently
else:
    try:
        wrapper.waiting_for_transfer_complete(statuses)
    except TransferError as e:
        logger.error("RDMA transfers failed after retry: %s", e)
        raise

def is_retryable_sq_full(status) -> bool:
    """Transient RDMA send-queue-full rejection, safe to re-post."""
    try:
        return bool(status.Failed()) and "SQ full" in (status.Message() or "")
    except Exception:
        return False

# before waiting, split statuses so retries are possible:
retryable = [s for s in statuses if is_retryable_sq_full(s)]
hard_failed = [s for s in statuses if s.Failed() and not is_retryable_sq_full(s)]

from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_engine import TransferError

try:
    wrapper.waiting_for_transfer_complete(statuses)
except TransferError as e:
    msg = str(e)
    if "timed out" in msg or "SQ full" in msg:
        logger.warning("Transient RDMA failure, backing off and retrying batch: %s", msg)
        time.sleep(backoff)
        statuses = wrapper.repost_transfers(request)  # re-issue the batch
        wrapper.waiting_for_transfer_complete(statuses)
    else:
        logger.error("Permanent RDMA transfer failure: %s", msg)
        raise

When it happens

Trigger: Calling waiting_for_transfer_complete(transfer_statuses) after read_remote_data/write_remote_data/write_remote_data_single (moriio_engine.py:622-654) and one or more statuses report Failed() — e.g. remote memory region invalidated, session torn down, 'SQ full' send-queue exhaustion — or the batch does not reach Succeeded() within transfer_timeout seconds (default from _transfer_timeout). High concurrency with many in-flight RDMA ops per request makes both SQ-full and timeout more likely.

Common situations: Disaggregated prefill/decode with bursty prefill traffic overloading the NIC send queue; transfer_timeout left at default while batch sizes or TP/DP width grew; remote decode worker crashing or restarting mid-transfer so completion never arrives (surfaces as timeout); fabric-level issues — flaky RoCE link, PFC storm, wrong GID/GRH config; stale remote memory descriptor after the peer re-registered its KV cache.

Related errors

• JSON error: {0}
• HTTP request failed: {0}
• tokenize endpoint unavailable: {0}
• Invalid MoRIIO backend {backend!r} in kv_connector_extra_con
• IO error: {0}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/7dd910c56ee553a5. Report an issue: GitHub.