langchain-ai/langchain · error · ImportError

Could not import transformers python package. This is needed

Error message

Could not import transformers python package. This is needed in order to calculate get_token_ids. Please install it with `pip install transformers`.

What it means

`ImportError` from `get_token_ids` in `langchain_core.language_models.base`: the optional `transformers` package is not installed, so the GPT-2 tokenizer (`GPT2TokenizerFast.from_pretrained('gpt2')`) cannot be created. This path is used to count/encode tokens with GPT-2, typically by `get_num_tokens` on base models.

Source

Thrown at libs/core/langchain_core/language_models/base.py:96

def get_tokenizer() -> Any:
    """Get a GPT-2 tokenizer instance.

    This function is cached to avoid re-loading the tokenizer every time it is called.

    Raises:
        ImportError: If the transformers package is not installed.

    Returns:
        The GPT-2 tokenizer instance.

    """
    if not _HAS_TRANSFORMERS:
        msg = (
            "Could not import transformers python package. "
            "This is needed in order to calculate get_token_ids. "
            "Please install it with `pip install transformers`."
        )
        raise ImportError(msg)
    # create a GPT-2 tokenizer instance
    return GPT2TokenizerFast.from_pretrained("gpt2")


_GPT2_TOKENIZER_WARNED = False


def _get_token_ids_default_method(text: str) -> list[int]:
    """Encode the text into token IDs using the fallback GPT-2 tokenizer."""
    global _GPT2_TOKENIZER_WARNED  # noqa: PLW0603
    if not _GPT2_TOKENIZER_WARNED:
        warnings.warn(
            "Using fallback GPT-2 tokenizer for token counting. "
            "Token counts may be inaccurate for non-GPT-2 models. "
            "For accurate counts, use a model-specific method if available.",
            stacklevel=3,
        )
        _GPT2_TOKENIZER_WARNED = True

View on GitHub (pinned to e32fa9a52e)

Solutions

Install the extra: `pip install transformers` (or add it to your dependency list / `langchain[transformers]`-style extras where available).
Prefer `get_num_tokens_from_messages` / tiktoken-based counting if your model family is OpenAI, or pass a custom `get_token_ids` callable to the model to avoid the transformers dependency entirely.
If you don't need token counts, remove the `get_num_tokens` call from your code path (e.g. estimate by characters instead).

Example fix

# before
tokens = llm.get_num_tokens(prompt)  # ImportError without transformers

# after
# shell: pip install transformers
tokens = llm.get_num_tokens(prompt)

# or supply a tiktoken-based counter
import tiktoken
enc = tiktoken.get_encoding("gpt2")
llm.get_token_ids = lambda text: enc.encode(text)
tokens = len(llm.get_token_ids(prompt))

Defensive patterns

Strategy: fallback

Validate before calling

from importlib.util import find_spec

HAS_TRANSFORMERS = find_spec("transformers") is not None
assert HAS_TRANSFORMERS, "pip install transformers for GPT-2 token counting"

Type guard

null

Try / catch

from importlib.util import find_spec

if find_spec("transformers") is None:
    # fallback: tiktoken GPT-2 encoding, no transformers needed
    import tiktoken
    _enc = tiktoken.get_encoding("gpt2")
    get_token_ids = lambda text: _enc.encode(text)  # noqa: E731
else:
    from langchain_core.language_models.base import get_token_ids

Prevention

Add transformers to deployment dependencies if any code path counts tokens
Cache the tokenizer (get_token_ids re-instantiates GPT2TokenizerFast per design; memoize it yourself for hot paths)
Consider tiktoken for OpenAI-family models — lighter dependency, same BPE

When it happens

Trigger: Calling `llm.get_num_tokens(text)` or `get_token_ids(text)` on a base `BaseLanguageModel`/`BaseLLM` in an environment where `transformers` is not installed (`_HAS_TRANSFORMERS` is False).

Common situations: Slim Docker images or CI environments installing only `langchain-core` without extras; deploying a server that only streams text but a code path (rate limiting, budgeting by tokens) calls `get_num_tokens`; assuming `transformers` is a hard dependency when it is optional.

Related errors

AI-assisted analysis of langchain-ai/langchain@e32fa9a52e (2026-08-14). Data as JSON: /api/errors/2bae5817b7d57d85. Report an issue: GitHub.