run-llama/llama_index · error · ValueError

`transformers` package not found, please run `pip install tr

Error message

`transformers` package not found, please run `pip install transformers`

What it means

Thrown by `get_transformer_tokenizer_fn` in llama-index-core/utils.py when the optional `transformers` package is not installed in the environment. LlamaIndex keeps transformers as an optional dependency (note the `pants: no-infer-dep` marker), so the tokenizer helper only works after an explicit install. The ImportError is converted into a ValueError with an actionable install hint.

Source

Thrown at llama-index-core/llama_index/core/utils.py:429

def count_tokens(text: str) -> int:
    tokenizer = get_tokenizer()
    tokens = tokenizer(text)
    return len(tokens)


def get_transformer_tokenizer_fn(model_name: str) -> Callable[[str], List[str]]:
    """
    Args:
        model_name(str): the model name of the tokenizer.
                        For instance, fxmarty/tiny-llama-fast-tokenizer.

    """
    try:
        from transformers import AutoTokenizer  # pants: no-infer-dep
    except ImportError:
        raise ValueError(
            "`transformers` package not found, please run `pip install transformers`"
        )
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    return tokenizer.tokenize


def get_cache_dir() -> str:
    """
    Locate a platform-appropriate cache directory for llama_index,
    and create it if it doesn't yet exist.
    """
    # User override
    if "LLAMA_INDEX_CACHE_DIR" in os.environ:
        path = Path(os.environ["LLAMA_INDEX_CACHE_DIR"])
    else:
        path = Path(platformdirs.user_cache_dir("llama_index"))

    # Pass exist_ok and call makedirs directly, so we avoid TOCTOU issues

View on GitHub (pinned to afd0fef371)

Solutions

1. Run `pip install transformers` (or add it to your project's dependencies).
2. Alternatively install the extra: `pip install llama-index-core[transformers]`.
3. If you do not need a HuggingFace tokenizer, fall back to a built-in splitter (e.g. SentenceSplitter) that does not require transformers.

Example fix

# before
fn = get_transformer_tokenizer_fn("fxmarty/tiny-llama-fast-tokenizer")  # ValueError

# after
# pip install transformers
fn = get_transformer_tokenizer_fn("fxmarty/tiny-llama-fast-tokenizer")

def transformers_available() -> bool:
    try:
        import transformers  # noqa: F401
        return True
    except ImportError:
        return False

if not transformers_available():
    raise SystemExit("Install with: pip install transformers")

try:
    fn = get_transformer_tokenizer_fn(model_name)
except ValueError as e:
    if "transformers" in str(e):
        # degrade to a built-in splitter that needs no tokenizer
        splitter = SentenceSplitter()

When it happens

Trigger: Calling `get_transformer_tokenizer_fn(model_name)` (used by tokenizer-based node parsers / text splitters such as `TokenizerAwareNodeParser` configured with a HuggingFace tokenizer name) in an environment where `import transformers` fails.

Common situations: Installing only `llama-index-core` (or the full `llama-index` meta-package) without extras like `llama-index-core[transformers]`; slim Docker images that strip optional deps; CI environments where the tokenizer-based splitter test runs without the extra installed.

Related errors

• `tiktoken` package not found, please run `pip install tiktok
• Invalid Embedding name: {name}
• `llama-index-embeddings-openai` package not found, please ru
• `llama-index-embeddings-clip` package not found, please run
• `llama-index-embeddings-huggingface` package not found, plea

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/bbafbb93cef52a47. Report an issue: GitHub.