vllm-project/vllm · error · Error

tokenizer error: {0}

Error message

tokenizer error: {0}

What it means

The Tokenizer(String) variant of vllm_text's Error enum, wrapping tokenizer failures as free-form text. It is produced by From<vllm_tokenizer::TokenizerError> and directly in the HF backend (backend/hf/config.rs, backend/hf/model_files.rs) for tokenizer config errors and model-file resolution failures — e.g. missing tokenizer files, bad tokenizer_config.json, or HF Hub download/API errors.

Source

Thrown at rust/src/text/src/error.rs:14

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project

use thiserror::Error;
use vllm_engine_core_client::Error as EngineCoreError;
use vllm_llm::Error as LlmError;

pub use crate::lower::logprobs::LogprobsError;
pub use crate::lower::sampling::SamplingParamsError;
pub use crate::lower::token_ids::TokenIdsError;

#[derive(Debug, Error)]
pub enum Error {
    #[error("tokenizer error: {0}")]
    Tokenizer(String),
    #[error("text request `{request_id}` must contain at least one prompt token ID")]
    EmptyPromptTokenIds { request_id: String },
    #[error("text request `{request_id}` stop strings cannot be empty")]
    EmptyStopString { request_id: String },
    #[error(
        "this model's maximum context length is {max_model_len} tokens, \
         but the prompt contains {prompt_len} input tokens"
    )]
    PromptTooLong { max_model_len: u32, prompt_len: u32 },
    #[error(transparent)]
    Logprobs(#[from] LogprobsError),
    #[error(transparent)]
    TokenIds(#[from] TokenIdsError),
    #[error(transparent)]
    SamplingParams(#[from] SamplingParamsError),
    #[error(
        "`min_tokens` must be less than or equal to `max_tokens`, \

View on GitHub (pinned to c794754062)

Solutions

1. Verify the model repo contains tokenizer files (tokenizer.json / tokenizer.model, tokenizer_config.json, config.json).
2. For gated/private models, set a valid HF token (HF_TOKEN) and confirm access.
3. Clear or repair the HF cache (HF_HOME) if a previous download was interrupted, and retry with network access.
4. If the tokenizer_config.json was hand-edited, restore the original — malformed JSON or unknown fields surface here.

Example fix

# before: repo without tokenizer files
vllm serve org/base-weights-only

# after
vllm serve org/model-with-tokenizer
# ensure network + token:
export HF_TOKEN=... && vllm serve gated/model

// before serving, confirm the tokenizer files resolve:
let files = std::fs::read_dir(hf_cache_dir.join("models--", model))
    .filter(|_| tokenizer_config_exists(model));
assert!(tokenizer_files_present(model), "model lacks tokenizer files");
export HF_TOKEN=...  # and verify: curl -I https://huggingface.co/<model>

fn tokenizer_files_present(model_dir: &Path) -> bool {
    ["tokenizer_config.json"].iter().all(|f| model_dir.join(f).exists())
        && ["tokenizer.json", "tokenizer.model"].iter().any(|f| model_dir.join(f).exists())
}

match result {
    Err(vllm_text::Error::Tokenizer(msg)) => {
        eprintln!("tokenizer setup failed: {msg}");
        // actionable branches: missing files -> fix repo; auth -> refresh HF_TOKEN; network -> retry offline cache
    }
    other => other,
}

When it happens

Trigger: Loading a model whose repo lacks tokenizer files; corrupt or incompatible tokenizer_config.json; HF Hub network/auth failure when resolving tokenizer files (build_api / repo download paths in model_files.rs); a resolved tokenizer file with no parent directory; unknown tokenizer type in config.

Common situations: Pointing --model at a repo with only safetensors (no tokenizer); offline environments without HF_HOME cache and blocked network; expired or missing HF token for gated models; partial/interrupted downloads leaving malformed JSON.

Related errors

• tokenizer error: {0}
• this model's maximum context length is {max_model_len} token
• token_id(s) {token_ids:?} in {parameter} are out of vocabula
• this model's maximum context length is {max_model_len} token
• transport error

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/db16bf1760ad8c66. Report an issue: GitHub.