huggingface/transformers · error · ValueError

Converting from SentencePiece and Tiktoken failed, if a conv

Error message

Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast converters: {list(SLOW_TO_FAST_CONVERTERS.keys())}

What it means

ValueError from convert_slow_tokenizer's fallback path: the tokenizer looked like a tiktoken candidate, but TikTokenConverter raised internally (missing tiktoken, missing/unreadable vocab_file, malformed ranks), and no SentencePiece converter applied either. The message lists all registered SLOW_TO_FAST converters as a hint.

Source

Thrown at src/transformers/convert_slow_tokenizer.py:2077

        converter_class = SLOW_TO_FAST_CONVERTERS[tokenizer_class_name]
        return converter_class(transformer_tokenizer).converted()

    vocab_file = transformer_tokenizer.vocab_file
    if isinstance(vocab_file, str) and os.path.isfile(vocab_file) and is_tekken_vocab_filename(vocab_file):
        from .integrations.mistral.tokenizer import MistralConverter

        transformer_tokenizer.original_tokenizer = transformer_tokenizer
        logger.info("Converting from Mistral tekken.json")
        return MistralConverter(vocab_file).converted()
    else:
        try:
            logger.info("Converting from Tiktoken")
            return TikTokenConverter(
                vocab_file=transformer_tokenizer.vocab_file,
                extra_special_tokens=transformer_tokenizer.extra_special_tokens,
            ).converted()
        except Exception:
            raise ValueError(
                f"Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path "
                f"with a SentencePiece tokenizer.model file."
                f"Currently available slow->fast converters: {list(SLOW_TO_FAST_CONVERTERS.keys())}"
            )

View on GitHub (pinned to a597f97485)

Solutions

1. Ensure the directory contains a valid tiktoken vocab file at the expected path (transformer_tokenizer.vocab_file) and install tiktoken.
2. If the tokenizer is sentencepiece-based, include the tokenizer.model file so the SentencePiece converter applies.
3. Check for an already-converted tokenizer.json and load it directly instead of converting.
4. As a fallback, load with use_fast=False.

Example fix

// before
AutoTokenizer.from_pretrained("./my_tiktoken_dir", use_fast=True)  # ValueError

// after
# ensure ./my_tiktoken_dir contains the tiktoken vocab file, then
# pip install tiktoken
AutoTokenizer.from_pretrained("./my_tiktoken_dir", use_fast=True)

vf = getattr(transformer_tokenizer, "vocab_file", None)
assert vf and os.path.exists(vf), f"vocab file missing or not found at {vf!r}"
try:
    from tiktoken.load import load_tiktoken_bpe  # noqa
except Exception:
    raise RuntimeError("install tiktoken or provide a sentencepiece tokenizer.model file")

def tokenizer_dir_is_convertible(d: str) -> bool:
    return any(os.path.exists(os.path.join(d, f)) for f in ("tokenizer.json", "tokenizer.model", "tiktoken vocab"))

try:
    tok = AutoTokenizer.from_pretrained(path, use_fast=True)
except ValueError as e:
    if "Converting from SentencePiece and Tiktoken failed" in str(e):
        tok = AutoTokenizer.from_pretrained(path, use_fast=False)

When it happens

Trigger: AutoTokenizer.from_pretrained(dir_with_tiktoken_file, use_fast=True) where vocab_file is None or not a valid tiktoken BPE file; or tiktoken not installed; or a vocab file in neither sentencepiece nor tiktoken format reaching the last-resort branch.

Common situations: Partial tokenizer directories (tokenizer.json absent, vocab file renamed), custom vocab formats, missing optional dependency chains, or a transformers_file (tekken.json) path not matching either converter.

Related errors

• `tiktoken` is required to read a `tiktoken` file. Install it
• {error_message} requires the protobuf library but it was not
• You're trying to run a `Unigram` model but you're file was t
• Unrecognized tokenizer name, should be one of {list(TOKENIZE
• Invalid checkpoint path: '{checkpoint}' attempts to escape `

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/07b45e4276538bcc. Report an issue: GitHub.