{"record":{"id":"bbafbb93cef52a47","repo":"run-llama/llama_index","slug":"transformers-package-not-found-please-run-pip","errorCode":null,"errorMessage":"`transformers` package not found, please run `pip install transformers`","messagePattern":"`transformers` package not found, please run `pip install transformers`","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/utils.py","lineNumber":429,"sourceCode":"\n\ndef count_tokens(text: str) -> int:\n    tokenizer = get_tokenizer()\n    tokens = tokenizer(text)\n    return len(tokens)\n\n\ndef get_transformer_tokenizer_fn(model_name: str) -> Callable[[str], List[str]]:\n    \"\"\"\n    Args:\n        model_name(str): the model name of the tokenizer.\n                        For instance, fxmarty/tiny-llama-fast-tokenizer.\n\n    \"\"\"\n    try:\n        from transformers import AutoTokenizer  # pants: no-infer-dep\n    except ImportError:\n        raise ValueError(\n            \"`transformers` package not found, please run `pip install transformers`\"\n        )\n    tokenizer = AutoTokenizer.from_pretrained(model_name)\n    return tokenizer.tokenize\n\n\ndef get_cache_dir() -> str:\n    \"\"\"\n    Locate a platform-appropriate cache directory for llama_index,\n    and create it if it doesn't yet exist.\n    \"\"\"\n    # User override\n    if \"LLAMA_INDEX_CACHE_DIR\" in os.environ:\n        path = Path(os.environ[\"LLAMA_INDEX_CACHE_DIR\"])\n    else:\n        path = Path(platformdirs.user_cache_dir(\"llama_index\"))\n\n    # Pass exist_ok and call makedirs directly, so we avoid TOCTOU issues","sourceCodeStart":411,"sourceCodeEnd":447,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/utils.py#L411-L447","documentation":"Thrown by `get_transformer_tokenizer_fn` in llama-index-core/utils.py when the optional `transformers` package is not installed in the environment. LlamaIndex keeps transformers as an optional dependency (note the `pants: no-infer-dep` marker), so the tokenizer helper only works after an explicit install. The ImportError is converted into a ValueError with an actionable install hint.","triggerScenarios":"Calling `get_transformer_tokenizer_fn(model_name)` (used by tokenizer-based node parsers / text splitters such as `TokenizerAwareNodeParser` configured with a HuggingFace tokenizer name) in an environment where `import transformers` fails.","commonSituations":"Installing only `llama-index-core` (or the full `llama-index` meta-package) without extras like `llama-index-core[transformers]`; slim Docker images that strip optional deps; CI environments where the tokenizer-based splitter test runs without the extra installed.","solutions":["Run `pip install transformers` (or add it to your project's dependencies).","Alternatively install the extra: `pip install llama-index-core[transformers]`.","If you do not need a HuggingFace tokenizer, fall back to a built-in splitter (e.g. SentenceSplitter) that does not require transformers."],"exampleFix":"# before\nfn = get_transformer_tokenizer_fn(\"fxmarty/tiny-llama-fast-tokenizer\")  # ValueError\n\n# after\n# pip install transformers\nfn = get_transformer_tokenizer_fn(\"fxmarty/tiny-llama-fast-tokenizer\")","handlingStrategy":"validation","validationCode":"def transformers_available() -> bool:\n    try:\n        import transformers  # noqa: F401\n        return True\n    except ImportError:\n        return False\n\nif not transformers_available():\n    raise SystemExit(\"Install with: pip install transformers\")","typeGuard":null,"tryCatchPattern":"try:\n    fn = get_transformer_tokenizer_fn(model_name)\nexcept ValueError as e:\n    if \"transformers\" in str(e):\n        # degrade to a built-in splitter that needs no tokenizer\n        splitter = SentenceSplitter()","preventionTips":["Declare transformers in project dependencies if any tokenizer-based splitter is used.","Fail fast at startup with an import probe rather than at first call.","Pin the transformers version to avoid tokenizer API drift."],"tags":["dependencies","optional-import","tokenizer","python"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}