vllm-project/vllm · error · TokenizerError

tokenizer error: {0}

Error message

tokenizer error: {0}

What it means

Generic wrapper (`TokenizerError` in rust/src/tokenizer/src/error.rs) around any failure while loading or running the Hugging Face tokenizer: file not found, malformed tokenizer.json, invalid model id, encode/decode failures. The underlying detail is carried in the string payload after the `tokenizer error:` prefix; the thiserror-ext Macro derives constructor helpers.

Source

Thrown at rust/src/tokenizer/src/error.rs:11

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project

use thiserror::Error;
use thiserror_ext::Macro;

pub type Result<T> = std::result::Result<T, TokenizerError>;

#[derive(Debug, Error, Macro)]
#[thiserror_ext(macro(path = "crate::error"))]
#[error("tokenizer error: {0}")]
pub struct TokenizerError(#[message] pub String);

View on GitHub (pinned to c794754062)

Solutions

1. Verify the model id/path exists and its tokenizer files download (test with `huggingface-cli download <model> --include "*token*"`)
2. In offline environments, pre-populate the HF cache or point HF_HOME at a local snapshot
3. Inspect the wrapped message string for the underlying cause (404, parse error, invalid model)

Example fix

# before
--model qwen/qwen2.5-7b-invla  # typo in org/name

# after
--model Qwen/Qwen2.5-7B-Instruct

// smoke-test the tokenizer source before startup
let path = std::path::Path::new(&model_path).join("tokenizer.json");
if model_is_local { assert!(path.exists(), "missing tokenizer.json at {path:?}"); }

match tokenizer::load(&model_id) {
    Err(e) => {
        tracing::error!(%model_id, error = %e.0, "tokenizer load failed");
        startup.abort();
    }
    Ok(tok) => tok,
}

When it happens

Trigger: Loading a tokenizer from a nonexistent hub id or local path, a corrupted/mismatched tokenizer.json, encoding text with characters the tokenizer cannot handle, or a revision/checkout where tokenizer files are missing.

Common situations: Wrong or misspelled model id; offline environments without HF cache or network access; tokenizer files out of sync with the model snapshot; LFS pointers downloaded instead of real weights/tokenizer files.

Related errors

• tokenizer error: {0}
• this model's maximum context length is {max_model_len} token
• token_id(s) {token_ids:?} in {parameter} are out of vocabula
• this model's maximum context length is {max_model_len} token
• tokenizer is missing reasoning delimiter token `{token}`

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/cd823d177c890454. Report an issue: GitHub.