{"record":{"id":"07b45e4276538bcc","repo":"huggingface/transformers","slug":"converting-from-sentencepiece-and-tiktoken-failed","errorCode":null,"errorMessage":"Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast converters: {list(SLOW_TO_FAST_CONVERTERS.keys())}","messagePattern":"Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer\\.model file\\.Currently available slow->fast converters: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/convert_slow_tokenizer.py","lineNumber":2077,"sourceCode":"        converter_class = SLOW_TO_FAST_CONVERTERS[tokenizer_class_name]\n        return converter_class(transformer_tokenizer).converted()\n\n    vocab_file = transformer_tokenizer.vocab_file\n    if isinstance(vocab_file, str) and os.path.isfile(vocab_file) and is_tekken_vocab_filename(vocab_file):\n        from .integrations.mistral.tokenizer import MistralConverter\n\n        transformer_tokenizer.original_tokenizer = transformer_tokenizer\n        logger.info(\"Converting from Mistral tekken.json\")\n        return MistralConverter(vocab_file).converted()\n    else:\n        try:\n            logger.info(\"Converting from Tiktoken\")\n            return TikTokenConverter(\n                vocab_file=transformer_tokenizer.vocab_file,\n                extra_special_tokens=transformer_tokenizer.extra_special_tokens,\n            ).converted()\n        except Exception:\n            raise ValueError(\n                f\"Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path \"\n                f\"with a SentencePiece tokenizer.model file.\"\n                f\"Currently available slow->fast converters: {list(SLOW_TO_FAST_CONVERTERS.keys())}\"\n            )\n","sourceCodeStart":2059,"sourceCodeEnd":2082,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/convert_slow_tokenizer.py#L2059-L2082","documentation":"ValueError from convert_slow_tokenizer's fallback path: the tokenizer looked like a tiktoken candidate, but TikTokenConverter raised internally (missing tiktoken, missing/unreadable vocab_file, malformed ranks), and no SentencePiece converter applied either. The message lists all registered SLOW_TO_FAST converters as a hint.","triggerScenarios":"AutoTokenizer.from_pretrained(dir_with_tiktoken_file, use_fast=True) where vocab_file is None or not a valid tiktoken BPE file; or tiktoken not installed; or a vocab file in neither sentencepiece nor tiktoken format reaching the last-resort branch.","commonSituations":"Partial tokenizer directories (tokenizer.json absent, vocab file renamed), custom vocab formats, missing optional dependency chains, or a transformers_file (tekken.json) path not matching either converter.","solutions":["Ensure the directory contains a valid tiktoken vocab file at the expected path (transformer_tokenizer.vocab_file) and install tiktoken.","If the tokenizer is sentencepiece-based, include the tokenizer.model file so the SentencePiece converter applies.","Check for an already-converted tokenizer.json and load it directly instead of converting.","As a fallback, load with use_fast=False."],"exampleFix":"// before\nAutoTokenizer.from_pretrained(\"./my_tiktoken_dir\", use_fast=True)  # ValueError\n\n// after\n# ensure ./my_tiktoken_dir contains the tiktoken vocab file, then\n# pip install tiktoken\nAutoTokenizer.from_pretrained(\"./my_tiktoken_dir\", use_fast=True)","handlingStrategy":"fallback","validationCode":"vf = getattr(transformer_tokenizer, \"vocab_file\", None)\nassert vf and os.path.exists(vf), f\"vocab file missing or not found at {vf!r}\"\ntry:\n    from tiktoken.load import load_tiktoken_bpe  # noqa\nexcept Exception:\n    raise RuntimeError(\"install tiktoken or provide a sentencepiece tokenizer.model file\")","typeGuard":"def tokenizer_dir_is_convertible(d: str) -> bool:\n    return any(os.path.exists(os.path.join(d, f)) for f in (\"tokenizer.json\", \"tokenizer.model\", \"tiktoken vocab\"))","tryCatchPattern":"try:\n    tok = AutoTokenizer.from_pretrained(path, use_fast=True)\nexcept ValueError as e:\n    if \"Converting from SentencePiece and Tiktoken failed\" in str(e):\n        tok = AutoTokenizer.from_pretrained(path, use_fast=False)","preventionTips":["Ship tokenizer.json alongside the slow files so conversion is never needed at load time.","Pre-install tiktoken and sentencepiece in environments that convert arbitrary checkpoints."],"tags":["tokenizer","conversion","tiktoken","fallback"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}