{"record":{"id":"cd823d177c890454","repo":"vllm-project/vllm","slug":"tokenizer-error-0-cd823d","errorCode":null,"errorMessage":"tokenizer error: {0}","messagePattern":"tokenizer error: (.+?)","errorType":"exception","errorClass":"TokenizerError","httpStatus":null,"severity":"error","filePath":"rust/src/tokenizer/src/error.rs","lineNumber":11,"sourceCode":"// SPDX-License-Identifier: Apache-2.0\n// SPDX-FileCopyrightText: Copyright contributors to the vLLM project\n\nuse thiserror::Error;\nuse thiserror_ext::Macro;\n\npub type Result<T> = std::result::Result<T, TokenizerError>;\n\n#[derive(Debug, Error, Macro)]\n#[thiserror_ext(macro(path = \"crate::error\"))]\n#[error(\"tokenizer error: {0}\")]\npub struct TokenizerError(#[message] pub String);\n","sourceCodeStart":1,"sourceCodeEnd":13,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/tokenizer/src/error.rs#L1-L13","documentation":"Generic wrapper (`TokenizerError` in rust/src/tokenizer/src/error.rs) around any failure while loading or running the Hugging Face tokenizer: file not found, malformed tokenizer.json, invalid model id, encode/decode failures. The underlying detail is carried in the string payload after the `tokenizer error:` prefix; the thiserror-ext Macro derives constructor helpers.","triggerScenarios":"Loading a tokenizer from a nonexistent hub id or local path, a corrupted/mismatched tokenizer.json, encoding text with characters the tokenizer cannot handle, or a revision/checkout where tokenizer files are missing.","commonSituations":"Wrong or misspelled model id; offline environments without HF cache or network access; tokenizer files out of sync with the model snapshot; LFS pointers downloaded instead of real weights/tokenizer files.","solutions":["Verify the model id/path exists and its tokenizer files download (test with `huggingface-cli download <model> --include \"*token*\"`)","In offline environments, pre-populate the HF cache or point HF_HOME at a local snapshot","Inspect the wrapped message string for the underlying cause (404, parse error, invalid model)"],"exampleFix":"# before\n--model qwen/qwen2.5-7b-invla  # typo in org/name\n\n# after\n--model Qwen/Qwen2.5-7B-Instruct","handlingStrategy":"try-catch","validationCode":"// smoke-test the tokenizer source before startup\nlet path = std::path::Path::new(&model_path).join(\"tokenizer.json\");\nif model_is_local { assert!(path.exists(), \"missing tokenizer.json at {path:?}\"); }","typeGuard":null,"tryCatchPattern":"match tokenizer::load(&model_id) {\n    Err(e) => {\n        tracing::error!(%model_id, error = %e.0, \"tokenizer load failed\");\n        startup.abort();\n    }\n    Ok(tok) => tok,\n}","preventionTips":["Validate model ids / HF connectivity in a preflight step at service start","Pin HF revisions and pre-download tokenizer files into HF_HOME for offline serving"],"tags":["rust","tokenizer","huggingface","vllm"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}