{"record":{"id":"7dd910c56ee553a5","repo":"vllm-project/vllm","slug":"tokenizer-error-0","errorCode":null,"errorMessage":"Tokenizer error: {0}","messagePattern":"Tokenizer error: (.+?)","errorType":"exception","errorClass":"BenchError","httpStatus":null,"severity":"critical","filePath":"rust/src/bench/src/error.rs","lineNumber":14,"sourceCode":"// SPDX-License-Identifier: Apache-2.0\n// SPDX-FileCopyrightText: Copyright contributors to the vLLM project\n\nuse thiserror::Error;\n\n#[derive(Error, Debug)]\npub enum BenchError {\n    #[error(\"HTTP request failed: {0}\")]\n    Http(#[from] reqwest::Error),\n\n    #[error(\"JSON error: {0}\")]\n    Json(#[from] serde_json::Error),\n\n    #[error(\"Tokenizer error: {0}\")]\n    Tokenizer(String),\n\n    /// The server's /tokenize//detokenize endpoint is not usable (4xx status:\n    /// not exposed, or rejected by a gateway such as LLM-d/EPP that returns\n    /// 400 instead of 404). Callers treat this as \"skip verification\", unlike\n    /// `Tokenizer` errors which are genuine failures.\n    #[error(\"tokenize endpoint unavailable: {0}\")]\n    TokenizeUnavailable(String),\n\n    #[error(\"Configuration error: {0}\")]\n    Config(String),\n\n    #[error(\"Endpoint not ready after {0}s: {1}\")]\n    EndpointTimeout(u64, String),\n\n    #[error(\"Backend error: {0}\")]\n    Backend(String),\n","sourceCodeStart":1,"sourceCodeEnd":32,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/bench/src/error.rs#L1-L32","documentation":"Raised by MoRIIOWrapper.waiting_for_transfer_complete (moriio_engine.py:696) after the wait loop finishes: it polls every RDMA transfer status (either the explicit transfer_statuses argument or all statuses drained from self.transfer_status) and collects two kinds of failures — terminal status.Failed() errors, and transfers still incomplete when the deadline (self._transfer_timeout, configurable via kv_connector_extra_config.transfer_timeout) expires. It then raises TransferError summarizing how many of the waited transfers failed and why. This is the single point where MoRIIO RDMA data-plane failures surface to the vLLM scheduler.","triggerScenarios":"Calling waiting_for_transfer_complete(transfer_statuses) after read_remote_data/write_remote_data/write_remote_data_single (moriio_engine.py:622-654) and one or more statuses report Failed() — e.g. remote memory region invalidated, session torn down, 'SQ full' send-queue exhaustion — or the batch does not reach Succeeded() within transfer_timeout seconds (default from _transfer_timeout). High concurrency with many in-flight RDMA ops per request makes both SQ-full and timeout more likely.","commonSituations":"Disaggregated prefill/decode with bursty prefill traffic overloading the NIC send queue; transfer_timeout left at default while batch sizes or TP/DP width grew; remote decode worker crashing or restarting mid-transfer so completion never arrives (surfaces as timeout); fabric-level issues — flaky RoCE link, PFC storm, wrong GID/GRH config; stale remote memory descriptor after the peer re-registered its KV cache.","solutions":["Read the per-transfer lines in the message: 'RDMA transfer failed: ... (code=...)' points at protocol/MR issues, while 'timed out after Ns' points at capacity/latency.","For timeouts, raise the budget via kv_connector_extra_config.transfer_timeout, and/or reduce transfer concurrency (fewer requests in flight, smaller batched merges) as the error text itself suggests.","For 'SQ full' style failures, reduce the number of concurrent batch_read/batch_write calls per step or batch blocks into fewer larger transfers via merge_contiguous_blocks.","Verify both endpoints are alive and healthy: a crashed/restarted peer produces indefinite timeouts; check decode/prefill engine logs and re-establish the session.","Check RDMA fabric health (ibstat, ibping, link errors) and that memory registrations on both sides are current (re-register after any KV cache resize)."],"exampleFix":"// before\nwrapper.waiting_for_transfer_complete(statuses)  # TransferError kills the step on any failure\n\n// after (retry once on transient SQ-full, then surface)\nfrom vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_engine import TransferError\n\nfor attempt in range(2):\n    failed = [s for s in statuses if s.Failed()]\n    if not failed:\n        break\n    statuses = wrapper.retry_failed_reads(failed)  # re-post reads that failed transiently\nelse:\n    try:\n        wrapper.waiting_for_transfer_complete(statuses)\n    except TransferError as e:\n        logger.error(\"RDMA transfers failed after retry: %s\", e)\n        raise","handlingStrategy":"retry","validationCode":null,"typeGuard":"def is_retryable_sq_full(status) -> bool:\n    \"\"\"Transient RDMA send-queue-full rejection, safe to re-post.\"\"\"\n    try:\n        return bool(status.Failed()) and \"SQ full\" in (status.Message() or \"\")\n    except Exception:\n        return False\n\n# before waiting, split statuses so retries are possible:\nretryable = [s for s in statuses if is_retryable_sq_full(s)]\nhard_failed = [s for s in statuses if s.Failed() and not is_retryable_sq_full(s)]","tryCatchPattern":"from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_engine import TransferError\n\ntry:\n    wrapper.waiting_for_transfer_complete(statuses)\nexcept TransferError as e:\n    msg = str(e)\n    if \"timed out\" in msg or \"SQ full\" in msg:\n        logger.warning(\"Transient RDMA failure, backing off and retrying batch: %s\", msg)\n        time.sleep(backoff)\n        statuses = wrapper.repost_transfers(request)  # re-issue the batch\n        wrapper.waiting_for_transfer_complete(statuses)\n    else:\n        logger.error(\"Permanent RDMA transfer failure: %s\", msg)\n        raise","preventionTips":["Set kv_connector_extra_config.transfer_timeout explicitly (do not rely on the default) sized to your worst-case batch.","Bound concurrent in-flight transfers per step; use merge_contiguous_blocks to coalesce blocks into fewer, larger RDMA ops.","Distinguish retryable statuses (SQ full) from terminal ones before waiting, mirroring the connector's _is_sq_full_status.","Monitor completion latency; a rising trend means the NIC queue depth or fabric is saturating before the timeout fires.","Treat persistent timeouts as a dead-peer signal: verify the remote engine is alive before retrying, or you will retry into a void."],"tags":["moriio","rdma","timeout","kv-transfer","vllm"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}