What causes "tokenize endpoint unavailable: {0}" in vllm-project/vllm?

Check the log line immediately preceding the HandshakeError — _handle_message logs 'Failed to decode msgpack message, will try as string' or 'Received non-UTF8 message' and the repr of the offending payload, which identifies the exact bad message.

vllm-project/vllm · error · BenchError

tokenize endpoint unavailable: {0}

Error message

tokenize endpoint unavailable: {0}

What it means

Logged then raised as HandshakeError in the _async_wait ZMQ ROUTER listener thread (moriio_engine.py:719) when _handle_message throws while processing an incoming notify message from a remote node. _handle_message first tries msgpack.loads for structured messages (remote_blocks / write_done / release) and falls back to UTF-8 string completions; unhandled formats raise MoRIIOError, and the structured handlers themselves raise MoRIIOError on bad payloads (empty block_notify_list, invalid consumer_tp_size) or AssertionError on role violations (e.g. decode receiving a release message). The listener converts any of these to HandshakeError, which kills the daemon notify thread — after which the node silently stops receiving transfer notifications.

Source

Thrown at rust/src/bench/src/error.rs:21

use thiserror::Error;

#[derive(Error, Debug)]
pub enum BenchError {
    #[error("HTTP request failed: {0}")]
    Http(#[from] reqwest::Error),

    #[error("JSON error: {0}")]
    Json(#[from] serde_json::Error),

    #[error("Tokenizer error: {0}")]
    Tokenizer(String),

    /// The server's /tokenize//detokenize endpoint is not usable (4xx status:
    /// not exposed, or rejected by a gateway such as LLM-d/EPP that returns
    /// 400 instead of 404). Callers treat this as "skip verification", unlike
    /// `Tokenizer` errors which are genuine failures.
    #[error("tokenize endpoint unavailable: {0}")]
    TokenizeUnavailable(String),

    #[error("Configuration error: {0}")]
    Config(String),

    #[error("Endpoint not ready after {0}s: {1}")]
    EndpointTimeout(u64, String),

    #[error("Backend error: {0}")]
    Backend(String),

    #[error("IO error: {0}")]
    Io(#[from] std::io::Error),
}

pub type Result<T> = std::result::Result<T, BenchError>;

View on GitHub (pinned to c794754062)

Solutions

Check the log line immediately preceding the HandshakeError — _handle_message logs 'Failed to decode msgpack message, will try as string' or 'Received non-UTF8 message' and the repr of the offending payload, which identifies the exact bad message.
Verify prefill and decode run the same vLLM/MoRIIO version so notify message schemas match; upgrade the older side.
Confirm role configuration is correct (one PRODUCER/prefill, one decode consumer) — role mismatches trip assertions inside the structured handlers.
Make sure nothing else (monitoring probe, scanner, stale process) can connect to the ZMQ notify port and inject non-protocol bytes.
Restart the affected engine process to restart the notify listener thread — it is a daemon thread and will not come back on its own after the raise.

Example fix

// before (any bad message kills the notify thread)
except Exception as e:
    logger.error("Error processing message: %s", e)
    raise HandshakeError(f"Error processing message: {e}") from e

// after (quarantine malformed messages, keep listener alive)
except Exception as e:
    logger.exception("Dropping malformed notify message from %s: %r", identity, msg)
    continue  # or count and abort after N consecutive failures

Defensive patterns

Strategy: validation

Validate before calling

import msgpack

KNOWN_MESSAGE_TYPES = {"remote_blocks", "write_done", "release"}

def is_valid_notify_payload(msg: bytes) -> bool:
    try:
        data = msgpack.loads(msg)
    except Exception:
        return True  # legacy bare-string completions are valid
    if not isinstance(data, dict):
        return False
    mtype = data.get("type", "remote_blocks" if "req_id" in data else None)
    return mtype in KNOWN_MESSAGE_TYPES and "transfer_id" in data

# e.g. gate before delegating to the engine's handler:
# if not is_valid_notify_payload(msg): log_and_drop(msg)

Type guard

from typing import TypeGuard, Any
import msgpack

def is_known_structured_message(msg: bytes) -> TypeGuard[dict[str, Any]] | bool:
    try:
        data = msgpack.loads(msg)
    except Exception:
        return False
    if not isinstance(data, dict):
        return False
    t = data.get("type", "remote_blocks" if "req_id" in data else None)
    return t in {"remote_blocks", "write_done", "release"}

Prevention

Keep prefill and decode instances on the same vLLM/MoRIIO version so the ZMQ notify protocol schema matches.
Restrict network access to the notify port (firewall/security group) so only known peers can connect; scanners produce unparseable bytes.
Verify role assignment (producer=prefill, consumer=decode) before starting; role asserts inside message handlers kill the listener thread.
Watch for the notify thread's death: after a HandshakeError the daemon thread exits and no further notifications arrive — alert on missing notify activity rather than waiting for timeouts downstream.
If you control a fork of the engine, consider dropping-and-counting malformed messages instead of re-raising, so one bad frame cannot silence the node.

When it happens

Trigger: A remote peer sends a message this node cannot parse or refuses: msgpack payload with an unknown 'type' (version skew between prefill and decode MoRIIO builds), a 'remote_blocks' message arriving at a decode-role node or a 'release'/'write_done' at the wrong role (assertion inside _handle_*_message), a structured message with missing 'transfer_id', or garbage bytes that are neither valid msgpack nor UTF-8. Any single bad message terminates the notify thread via the 'raise' after logging.

Common situations: Prefill and decode instances running different vLLM/MoRIIO versions where the notify protocol gained new message types; a peer from an older deployment still sending bare-string completions after the node upgraded to structured-only handling; role misconfiguration (e.g. both sides set as producers) so role asserts fire; a non-MoRIIO process or port scanner connecting to the notify port and sending arbitrary bytes; message corruption over the wire.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/8293a49d1090d057. Report an issue: GitHub.