{"record":{"id":"db16bf1760ad8c66","repo":"vllm-project/vllm","slug":"tokenizer-error-0-db16bf","errorCode":null,"errorMessage":"tokenizer error: {0}","messagePattern":"tokenizer error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"rust/src/text/src/error.rs","lineNumber":14,"sourceCode":"// SPDX-License-Identifier: Apache-2.0\n// SPDX-FileCopyrightText: Copyright contributors to the vLLM project\n\nuse thiserror::Error;\nuse vllm_engine_core_client::Error as EngineCoreError;\nuse vllm_llm::Error as LlmError;\n\npub use crate::lower::logprobs::LogprobsError;\npub use crate::lower::sampling::SamplingParamsError;\npub use crate::lower::token_ids::TokenIdsError;\n\n#[derive(Debug, Error)]\npub enum Error {\n    #[error(\"tokenizer error: {0}\")]\n    Tokenizer(String),\n    #[error(\"text request `{request_id}` must contain at least one prompt token ID\")]\n    EmptyPromptTokenIds { request_id: String },\n    #[error(\"text request `{request_id}` stop strings cannot be empty\")]\n    EmptyStopString { request_id: String },\n    #[error(\n        \"this model's maximum context length is {max_model_len} tokens, \\\n         but the prompt contains {prompt_len} input tokens\"\n    )]\n    PromptTooLong { max_model_len: u32, prompt_len: u32 },\n    #[error(transparent)]\n    Logprobs(#[from] LogprobsError),\n    #[error(transparent)]\n    TokenIds(#[from] TokenIdsError),\n    #[error(transparent)]\n    SamplingParams(#[from] SamplingParamsError),\n    #[error(\n        \"`min_tokens` must be less than or equal to `max_tokens`, \\","sourceCodeStart":1,"sourceCodeEnd":32,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/text/src/error.rs#L1-L32","documentation":"The Tokenizer(String) variant of vllm_text's Error enum, wrapping tokenizer failures as free-form text. It is produced by From<vllm_tokenizer::TokenizerError> and directly in the HF backend (backend/hf/config.rs, backend/hf/model_files.rs) for tokenizer config errors and model-file resolution failures — e.g. missing tokenizer files, bad tokenizer_config.json, or HF Hub download/API errors.","triggerScenarios":"Loading a model whose repo lacks tokenizer files; corrupt or incompatible tokenizer_config.json; HF Hub network/auth failure when resolving tokenizer files (build_api / repo download paths in model_files.rs); a resolved tokenizer file with no parent directory; unknown tokenizer type in config.","commonSituations":"Pointing --model at a repo with only safetensors (no tokenizer); offline environments without HF_HOME cache and blocked network; expired or missing HF token for gated models; partial/interrupted downloads leaving malformed JSON.","solutions":["Verify the model repo contains tokenizer files (tokenizer.json / tokenizer.model, tokenizer_config.json, config.json).","For gated/private models, set a valid HF token (HF_TOKEN) and confirm access.","Clear or repair the HF cache (HF_HOME) if a previous download was interrupted, and retry with network access.","If the tokenizer_config.json was hand-edited, restore the original — malformed JSON or unknown fields surface here."],"exampleFix":"# before: repo without tokenizer files\nvllm serve org/base-weights-only\n\n# after\nvllm serve org/model-with-tokenizer\n# ensure network + token:\nexport HF_TOKEN=... && vllm serve gated/model","handlingStrategy":"validation","validationCode":"// before serving, confirm the tokenizer files resolve:\nlet files = std::fs::read_dir(hf_cache_dir.join(\"models--\", model))\n    .filter(|_| tokenizer_config_exists(model));\nassert!(tokenizer_files_present(model), \"model lacks tokenizer files\");\nexport HF_TOKEN=...  # and verify: curl -I https://huggingface.co/<model>","typeGuard":"fn tokenizer_files_present(model_dir: &Path) -> bool {\n    [\"tokenizer_config.json\"].iter().all(|f| model_dir.join(f).exists())\n        && [\"tokenizer.json\", \"tokenizer.model\"].iter().any(|f| model_dir.join(f).exists())\n}","tryCatchPattern":"match result {\n    Err(vllm_text::Error::Tokenizer(msg)) => {\n        eprintln!(\"tokenizer setup failed: {msg}\");\n        // actionable branches: missing files -> fix repo; auth -> refresh HF_TOKEN; network -> retry offline cache\n    }\n    other => other,\n}","preventionTips":["Pre-download the model (huggingface-cli download) in a separate CI step so network failures surface there.","Pin a valid HF_TOKEN for gated repos and rotate it before expiry.","Clear partial HF cache entries after interrupted downloads."],"tags":["tokenizer","huggingface","model-loading","network","rust","vllm"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}