{"record":{"id":"e150352a23dc9288","repo":"huggingface/transformers","slug":"tiktoken-is-required-to-read-a-tiktoken-file","errorCode":null,"errorMessage":"`tiktoken` is required to read a `tiktoken` file. Install it with `pip install tiktoken`.","messagePattern":"`tiktoken` is required to read a `tiktoken` file\\. Install it with `pip install tiktoken`\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/convert_slow_tokenizer.py","lineNumber":1927,"sourceCode":"        self,\n        vocab_file=None,\n        pattern=r\"\"\"(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+\"\"\",\n        add_prefix_space=False,\n        extra_special_tokens=None,\n        **kwargs,\n    ):\n        self.vocab_file = vocab_file\n        self.pattern = pattern\n        self.add_prefix_space = add_prefix_space\n        self.extra_special_tokens = (\n            extra_special_tokens.keys() if isinstance(extra_special_tokens, dict) else extra_special_tokens\n        )\n\n    def extract_vocab_merges_from_model(self, tiktoken_url: str):\n        try:\n            from tiktoken.load import load_tiktoken_bpe\n        except Exception:\n            raise ValueError(\n                \"`tiktoken` is required to read a `tiktoken` file. Install it with `pip install tiktoken`.\"\n            )\n\n        bpe_ranks = load_tiktoken_bpe(tiktoken_url)\n        byte_encoder = bytes_to_unicode()\n\n        def token_bytes_to_string(b):\n            return \"\".join([byte_encoder[ord(char)] for char in b.decode(\"latin-1\")])\n\n        merges = []\n        vocab = {}\n        for token, rank in bpe_ranks.items():\n            vocab[token_bytes_to_string(token)] = rank\n            if len(token) == 1:\n                continue\n            local = []\n            for index in range(1, len(token)):\n                piece_l, piece_r = token[:index], token[index:]","sourceCodeStart":1909,"sourceCodeEnd":1945,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/convert_slow_tokenizer.py#L1909-L1945","documentation":"ValueError from TikTokenConverter.extract_vocab_merges_from_model: importing load_tiktoken_bpe from the tiktoken package failed, so the BPE ranks of the tiktoken vocab file cannot be read. tiktoken is an optional dependency needed only for tiktoken-format (e.g. GPT-style /tiktoken) files.","triggerScenarios":"TikTokenConverter(vocab_file=\"...\").converted() or convert_slow_tokenizer on a tokenizer whose vocab file is in tiktoken format, in an environment where tiktoken is not installed or its install is broken.","commonSituations":"Slim production images, conversion scripts run in CI without optional deps, tiktoken installed for a different Python than the one running transformers.","solutions":["pip install tiktoken in the same interpreter/environment that runs transformers.","Verify with python -c \"from tiktoken.load import load_tiktoken_bpe\" before converting.","If you do not need tiktoken conversion, point the converter at a sentencepiece vocab file instead."],"exampleFix":"// before\nTikTokenConverter(vocab_file=\"gpt2.tiktoken\").converted()  # ValueError\n\n// after\n# pip install tiktoken\nTikTokenConverter(vocab_file=\"gpt2.tiktoken\").converted()","handlingStrategy":"validation","validationCode":"try:\n    from tiktoken.load import load_tiktoken_bpe  # noqa\n    tiktoken_ok = True\nexcept Exception:\n    tiktoken_ok = False\nif not tiktoken_ok:\n    raise RuntimeError(\"pip install tiktoken before tiktoken conversion\")","typeGuard":"def tiktoken_is_importable() -> bool:\n    try:\n        from tiktoken.load import load_tiktoken_bpe  # noqa: F401\n        return True\n    except Exception:\n        return False","tryCatchPattern":null,"preventionTips":["Include tiktoken in the environment for any workflow that touches GPT-style/tiktoken vocab files.","Prefer declarative extras: pip install 'transformers[tiktoken]' where supported."],"tags":["dependencies","tiktoken","tokenizer","conversion"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}