{"record":{"id":"594c166c22661e52","repo":"run-llama/llama_index","slug":"tiktoken-package-not-found-please-run-pip-inst","errorCode":null,"errorMessage":"`tiktoken` package not found, please run `pip install tiktoken`","messagePattern":"`tiktoken` package not found, please run `pip install tiktoken`","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/utils.py","lineNumber":163,"sourceCode":"    import llama_index.core\n\n    if isinstance(tokenizer, Tokenizer):\n        llama_index.core.global_tokenizer = tokenizer.encode\n    else:\n        llama_index.core.global_tokenizer = tokenizer\n\n\ndef get_tokenizer(model_name: str = \"gpt-3.5-turbo\") -> Callable[[str], List]:\n    import llama_index.core\n\n    if llama_index.core.global_tokenizer is None:\n        tiktoken_import_err = (\n            \"`tiktoken` package not found, please run `pip install tiktoken`\"\n        )\n        try:\n            import tiktoken\n        except ImportError:\n            raise ImportError(tiktoken_import_err)\n\n        # set tokenizer cache temporarily\n        should_revert = False\n        if \"TIKTOKEN_CACHE_DIR\" not in os.environ:\n            should_revert = True\n            os.environ[\"TIKTOKEN_CACHE_DIR\"] = os.path.join(\n                os.path.dirname(os.path.abspath(__file__)),\n                \"_static/tiktoken_cache\",\n            )\n\n        enc = tiktoken.encoding_for_model(model_name)\n        tokenizer = partial(enc.encode, allowed_special=\"all\")\n        set_global_tokenizer(tokenizer)\n\n        if should_revert:\n            del os.environ[\"TIKTOKEN_CACHE_DIR\"]\n\n    assert llama_index.core.global_tokenizer is not None","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/utils.py#L145-L181","documentation":"llama_index.core.utils.get_tokenizer() lazily imports tiktoken the first time a tokenizer is needed (default model gpt-3.5-turbo). tiktoken is an optional dependency of llama-index-core, so if it is missing the ImportError instructs you to install it. The function also configures a bundled TIKTOKEN_CACHE_DIR so encodings load offline.","triggerScenarios":"Any token counting (e.g. building an index, chunking with a token splitter, SentenceSplitter with default settings) in an environment without tiktoken, while llama_index.core.global_tokenizer is still None.","commonSituations":"Minimal installs of llama-index-core without extras; production images that trimmed 'heavy' packages; air-gapped environments where the wheel was never downloaded.","solutions":["Install it: pip install tiktoken.","Or avoid the dependency by setting a custom tokenizer first: llama_index.core.set_global_tokenizer(my_tokenizer).","For offline/air-gapped hosts, pre-populate TIKTOKEN_CACHE_DIR from a machine with network access."],"exampleFix":"# before (ImportError during chunking)\n splitter = SentenceSplitter(chunk_size=256)\n\n# after (shell)\n# pip install tiktoken\n\n# after (alternative: custom tokenizer)\nimport llama_index.core\nllama_index.core.set_global_tokenizer(lambda s: s.split())  # toy example","handlingStrategy":"validation","validationCode":"def tiktoken_available() -> bool:\n    try:\n        import tiktoken  # noqa: F401\n        return True\n    except ImportError:\n        return False\n\n# at startup: if not tiktoken_available(): llama_index.core.set_global_tokenizer(fallback)","typeGuard":"def has_tiktoken() -> bool:\n    try:\n        import tiktoken  # noqa: F401\n        return True\n    except ImportError:\n        return False","tryCatchPattern":"try:\n    splitter = SentenceSplitter(chunk_size=256)\nexcept ImportError as e:\n    if 'tiktoken' in str(e):\n        import llama_index.core\n        llama_index.core.set_global_tokenizer(lambda s: s.split()[:256])  # crude fallback\n        splitter = SentenceSplitter(chunk_size=256)\n    else:\n        raise","preventionTips":["Add tiktoken to requirements for any token-based chunking path.","Probe the import at startup and either install or set a global tokenizer.","For air-gapped hosts, pre-seed TIKTOKEN_CACHE_DIR from a networked machine."],"tags":["tokenizer","tiktoken","import-error","dependencies"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}