huggingface/transformers · error · ValueError

`tiktoken` is required to read a `tiktoken` file. Install it

Error message

`tiktoken` is required to read a `tiktoken` file. Install it with `pip install tiktoken`.

What it means

ValueError from TikTokenConverter.extract_vocab_merges_from_model: importing load_tiktoken_bpe from the tiktoken package failed, so the BPE ranks of the tiktoken vocab file cannot be read. tiktoken is an optional dependency needed only for tiktoken-format (e.g. GPT-style /tiktoken) files.

Source

Thrown at src/transformers/convert_slow_tokenizer.py:1927

        self,
        vocab_file=None,
        pattern=r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+""",
        add_prefix_space=False,
        extra_special_tokens=None,
        **kwargs,
    ):
        self.vocab_file = vocab_file
        self.pattern = pattern
        self.add_prefix_space = add_prefix_space
        self.extra_special_tokens = (
            extra_special_tokens.keys() if isinstance(extra_special_tokens, dict) else extra_special_tokens
        )

    def extract_vocab_merges_from_model(self, tiktoken_url: str):
        try:
            from tiktoken.load import load_tiktoken_bpe
        except Exception:
            raise ValueError(
                "`tiktoken` is required to read a `tiktoken` file. Install it with `pip install tiktoken`."
            )

        bpe_ranks = load_tiktoken_bpe(tiktoken_url)
        byte_encoder = bytes_to_unicode()

        def token_bytes_to_string(b):
            return "".join([byte_encoder[ord(char)] for char in b.decode("latin-1")])

        merges = []
        vocab = {}
        for token, rank in bpe_ranks.items():
            vocab[token_bytes_to_string(token)] = rank
            if len(token) == 1:
                continue
            local = []
            for index in range(1, len(token)):
                piece_l, piece_r = token[:index], token[index:]

View on GitHub (pinned to a597f97485)

Solutions

1. pip install tiktoken in the same interpreter/environment that runs transformers.
2. Verify with python -c "from tiktoken.load import load_tiktoken_bpe" before converting.
3. If you do not need tiktoken conversion, point the converter at a sentencepiece vocab file instead.

Example fix

// before
TikTokenConverter(vocab_file="gpt2.tiktoken").converted()  # ValueError

// after
# pip install tiktoken
TikTokenConverter(vocab_file="gpt2.tiktoken").converted()

try:
    from tiktoken.load import load_tiktoken_bpe  # noqa
    tiktoken_ok = True
except Exception:
    tiktoken_ok = False
if not tiktoken_ok:
    raise RuntimeError("pip install tiktoken before tiktoken conversion")

def tiktoken_is_importable() -> bool:
    try:
        from tiktoken.load import load_tiktoken_bpe  # noqa: F401
        return True
    except Exception:
        return False

When it happens

Trigger: TikTokenConverter(vocab_file="...").converted() or convert_slow_tokenizer on a tokenizer whose vocab file is in tiktoken format, in an environment where tiktoken is not installed or its install is broken.

Common situations: Slim production images, conversion scripts run in CI without optional deps, tiktoken installed for a different Python than the one running transformers.

Related errors

• {error_message} requires the protobuf library but it was not
• Converting from SentencePiece and Tiktoken failed, if a conv
• You're trying to run a `Unigram` model but you're file was t
• Unrecognized tokenizer name, should be one of {list(TOKENIZE
• You need to install optimum-quanto in order to use KV cache

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/e150352a23dc9288. Report an issue: GitHub.