{"record":{"id":"22973cf9292b0ef9","repo":"huggingface/candle","slug":"unsupported-tokenizer-model-model-kind","errorCode":null,"errorMessage":"unsupported tokenizer model `{model_kind}`","messagePattern":"unsupported tokenizer model `(.+?)`","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-core/src/quantized/tokenizer.rs","lineNumber":208,"sourceCode":"    let proc = tokenizers::processors::template::TemplateProcessing::builder()\n        .try_single(single)\n        .ok()?\n        .try_pair(pair)\n        .ok()?\n        .special_tokens(specials)\n        .build()\n        .ok()?;\n\n    Some(PostProcessorWrapper::Template(proc))\n}\n\nimpl TokenizerFromGguf for Tokenizer {\n    fn from_gguf(ct: &gguf_file::Content) -> Result<Self> {\n        let model_kind = metadata_value(ct, \"tokenizer.ggml.model\")?\n            .to_string()?\n            .to_lowercase();\n        if model_kind != \"gpt2\" {\n            crate::bail!(\"unsupported tokenizer model `{model_kind}`\");\n        }\n\n        let tokens = value_to_string_array(\n            metadata_value(ct, \"tokenizer.ggml.tokens\")?,\n            \"tokenizer.ggml.tokens\",\n        )?;\n        let vocab: Vocab = tokens\n            .iter()\n            .enumerate()\n            .map(|(i, t)| (t.clone(), i as u32))\n            .collect();\n        let merges = merges_from_value(metadata_value(ct, \"tokenizer.ggml.merges\")?)?;\n\n        let mut builder = BPE::builder().vocab_and_merges(vocab, merges);\n\n        if let Ok(val) = metadata_value(ct, \"tokenizer.ggml.unk_token_id\") {\n            let token_id = gguf_value_to_u32(val)?;\n            if let Some(token) = tokens.get(token_id as usize) {","sourceCodeStart":190,"sourceCodeEnd":226,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-core/src/quantized/tokenizer.rs#L190-L226","documentation":"Tokenizer::from_gguf only supports GGUF tokenizers whose tokenizer.ggml.model is \"gpt2\" (the BPE-style layout candle knows how to rebuild). Any other tokenizer model kind stored in the GGUF bails with this message.","triggerScenarios":"Calling Tokenizer::from_gguf (or QMatMul/model loaders that pull the tokenizer from GGUF content) on a file whose tokenizer.ggml.model metadata is not \"gpt2\", e.g. \"llama\" (sentencepiece/SPM) or \"rwkv\".","commonSituations":"Loading LLaMA-family GGUF models that use the SentencePiece tokenizer; older or alternative exporters writing non-gpt2 tokenizer models; expecting candle to auto-convert SPM tokenizers.","solutions":["Use a GGUF converted with a gpt2/BPE tokenizer, or convert the tokenizer to a standard tokenizer.json and load with tokenizers::Tokenizer::from_file instead","Re-export the model forcing tokenizer type gpt2 if the vocab is BPE-compatible","Extract tokens/merges from GGUF metadata yourself and construct the candle Tokenizer manually"],"exampleFix":"// before\nlet tok = Tokenizer::from_gguf(&ct)?;\n// after\nlet tok = tokenizers::Tokenizer::from_file(\"tokenizer.json\")?;","handlingStrategy":"validation","validationCode":"let kind = ct.metadata.get(\"tokenizer.ggml.model\")\n    .map(|v| v.to_string())\n    .transpose()?.unwrap_or_default().to_lowercase();\nif kind != \"gpt2\" {\n    // fall back to tokenizer.json instead of Tokenizer::from_gguf\n}","typeGuard":null,"tryCatchPattern":"match Tokenizer::from_gguf(&ct) {\n    Ok(t) => t,\n    Err(e) if e.to_string().contains(\"unsupported tokenizer model\") => {\n        tokenizers::Tokenizer::from_file(\"tokenizer.json\")?.into()\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Check tokenizer.ggml.model in GGUF metadata before using from_gguf","Ship a separate tokenizer.json alongside GGUF models with non-BPE tokenizers","Prefer loading the tokenizer from tokenizer.json for LLaMA/SPM models"],"tags":["candle","gguf","tokenizer","unsupported"],"backgroundTag":"unsupported-tokenizer-model","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}