{"record":{"id":"2bae5817b7d57d85","repo":"langchain-ai/langchain","slug":"could-not-import-transformers-python-package-this","errorCode":null,"errorMessage":"Could not import transformers python package. This is needed in order to calculate get_token_ids. Please install it with `pip install transformers`.","messagePattern":"Could not import transformers python package\\. This is needed in order to calculate get_token_ids\\. Please install it with `pip install transformers`\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"libs/core/langchain_core/language_models/base.py","lineNumber":96,"sourceCode":"def get_tokenizer() -> Any:\n    \"\"\"Get a GPT-2 tokenizer instance.\n\n    This function is cached to avoid re-loading the tokenizer every time it is called.\n\n    Raises:\n        ImportError: If the transformers package is not installed.\n\n    Returns:\n        The GPT-2 tokenizer instance.\n\n    \"\"\"\n    if not _HAS_TRANSFORMERS:\n        msg = (\n            \"Could not import transformers python package. \"\n            \"This is needed in order to calculate get_token_ids. \"\n            \"Please install it with `pip install transformers`.\"\n        )\n        raise ImportError(msg)\n    # create a GPT-2 tokenizer instance\n    return GPT2TokenizerFast.from_pretrained(\"gpt2\")\n\n\n_GPT2_TOKENIZER_WARNED = False\n\n\ndef _get_token_ids_default_method(text: str) -> list[int]:\n    \"\"\"Encode the text into token IDs using the fallback GPT-2 tokenizer.\"\"\"\n    global _GPT2_TOKENIZER_WARNED  # noqa: PLW0603\n    if not _GPT2_TOKENIZER_WARNED:\n        warnings.warn(\n            \"Using fallback GPT-2 tokenizer for token counting. \"\n            \"Token counts may be inaccurate for non-GPT-2 models. \"\n            \"For accurate counts, use a model-specific method if available.\",\n            stacklevel=3,\n        )\n        _GPT2_TOKENIZER_WARNED = True","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/langchain-ai/langchain/blob/e32fa9a52eab3b61ad7a45399bfde59b3e580fc4/libs/core/langchain_core/language_models/base.py#L78-L114","documentation":"`ImportError` from `get_token_ids` in `langchain_core.language_models.base`: the optional `transformers` package is not installed, so the GPT-2 tokenizer (`GPT2TokenizerFast.from_pretrained('gpt2')`) cannot be created. This path is used to count/encode tokens with GPT-2, typically by `get_num_tokens` on base models.","triggerScenarios":"Calling `llm.get_num_tokens(text)` or `get_token_ids(text)` on a base `BaseLanguageModel`/`BaseLLM` in an environment where `transformers` is not installed (`_HAS_TRANSFORMERS` is False).","commonSituations":"Slim Docker images or CI environments installing only `langchain-core` without extras; deploying a server that only streams text but a code path (rate limiting, budgeting by tokens) calls `get_num_tokens`; assuming `transformers` is a hard dependency when it is optional.","solutions":["Install the extra: `pip install transformers` (or add it to your dependency list / `langchain[transformers]`-style extras where available).","Prefer `get_num_tokens_from_messages` / tiktoken-based counting if your model family is OpenAI, or pass a custom `get_token_ids` callable to the model to avoid the transformers dependency entirely.","If you don't need token counts, remove the `get_num_tokens` call from your code path (e.g. estimate by characters instead)."],"exampleFix":"# before\ntokens = llm.get_num_tokens(prompt)  # ImportError without transformers\n\n# after\n# shell: pip install transformers\ntokens = llm.get_num_tokens(prompt)\n\n# or supply a tiktoken-based counter\nimport tiktoken\nenc = tiktoken.get_encoding(\"gpt2\")\nllm.get_token_ids = lambda text: enc.encode(text)\ntokens = len(llm.get_token_ids(prompt))","handlingStrategy":"fallback","validationCode":"from importlib.util import find_spec\n\nHAS_TRANSFORMERS = find_spec(\"transformers\") is not None\nassert HAS_TRANSFORMERS, \"pip install transformers for GPT-2 token counting\"","typeGuard":"null","tryCatchPattern":"from importlib.util import find_spec\n\nif find_spec(\"transformers\") is None:\n    # fallback: tiktoken GPT-2 encoding, no transformers needed\n    import tiktoken\n    _enc = tiktoken.get_encoding(\"gpt2\")\n    get_token_ids = lambda text: _enc.encode(text)  # noqa: E731\nelse:\n    from langchain_core.language_models.base import get_token_ids","preventionTips":["Add transformers to deployment dependencies if any code path counts tokens","Cache the tokenizer (get_token_ids re-instantiates GPT2TokenizerFast per design; memoize it yourself for hot paths)","Consider tiktoken for OpenAI-family models — lighter dependency, same BPE"],"tags":["dependencies","tokenizer","import-error"],"backgroundTag":null,"analyzedSha":"e32fa9a52eab3b61ad7a45399bfde59b3e580fc4","analyzedAt":"2026-08-14T18:42:09.092Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}