run-llama/llama_index · error · ImportError

`tiktoken` package not found, please run `pip install tiktok

Error message

`tiktoken` package not found, please run `pip install tiktoken`

What it means

llama_index.core.utils.get_tokenizer() lazily imports tiktoken the first time a tokenizer is needed (default model gpt-3.5-turbo). tiktoken is an optional dependency of llama-index-core, so if it is missing the ImportError instructs you to install it. The function also configures a bundled TIKTOKEN_CACHE_DIR so encodings load offline.

Source

Thrown at llama-index-core/llama_index/core/utils.py:163

    import llama_index.core

    if isinstance(tokenizer, Tokenizer):
        llama_index.core.global_tokenizer = tokenizer.encode
    else:
        llama_index.core.global_tokenizer = tokenizer


def get_tokenizer(model_name: str = "gpt-3.5-turbo") -> Callable[[str], List]:
    import llama_index.core

    if llama_index.core.global_tokenizer is None:
        tiktoken_import_err = (
            "`tiktoken` package not found, please run `pip install tiktoken`"
        )
        try:
            import tiktoken
        except ImportError:
            raise ImportError(tiktoken_import_err)

        # set tokenizer cache temporarily
        should_revert = False
        if "TIKTOKEN_CACHE_DIR" not in os.environ:
            should_revert = True
            os.environ["TIKTOKEN_CACHE_DIR"] = os.path.join(
                os.path.dirname(os.path.abspath(__file__)),
                "_static/tiktoken_cache",
            )

        enc = tiktoken.encoding_for_model(model_name)
        tokenizer = partial(enc.encode, allowed_special="all")
        set_global_tokenizer(tokenizer)

        if should_revert:
            del os.environ["TIKTOKEN_CACHE_DIR"]

    assert llama_index.core.global_tokenizer is not None

View on GitHub (pinned to afd0fef371)

Solutions

Install it: pip install tiktoken.
Or avoid the dependency by setting a custom tokenizer first: llama_index.core.set_global_tokenizer(my_tokenizer).
For offline/air-gapped hosts, pre-populate TIKTOKEN_CACHE_DIR from a machine with network access.

Example fix

# before (ImportError during chunking)
 splitter = SentenceSplitter(chunk_size=256)

# after (shell)
# pip install tiktoken

# after (alternative: custom tokenizer)
import llama_index.core
llama_index.core.set_global_tokenizer(lambda s: s.split())  # toy example

Defensive patterns

Strategy: validation

Validate before calling

def tiktoken_available() -> bool:
    try:
        import tiktoken  # noqa: F401
        return True
    except ImportError:
        return False

# at startup: if not tiktoken_available(): llama_index.core.set_global_tokenizer(fallback)

Type guard

def has_tiktoken() -> bool:
    try:
        import tiktoken  # noqa: F401
        return True
    except ImportError:
        return False

Try / catch

try:
    splitter = SentenceSplitter(chunk_size=256)
except ImportError as e:
    if 'tiktoken' in str(e):
        import llama_index.core
        llama_index.core.set_global_tokenizer(lambda s: s.split()[:256])  # crude fallback
        splitter = SentenceSplitter(chunk_size=256)
    else:
        raise

Prevention

Add tiktoken to requirements for any token-based chunking path.
Probe the import at startup and either install or set a global tokenizer.
For air-gapped hosts, pre-seed TIKTOKEN_CACHE_DIR from a networked machine.

When it happens

Trigger: Any token counting (e.g. building an index, chunking with a token splitter, SentenceSplitter with default settings) in an environment without tiktoken, while llama_index.core.global_tokenizer is still None.

Common situations: Minimal installs of llama-index-core without extras; production images that trimmed 'heavy' packages; air-gapped environments where the wheel was never downloaded.

Related errors

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/594c166c22661e52. Report an issue: GitHub.