chroma-core/chroma · error · ValueError

The tokenizers python package is not installed. Please insta

Error message

The tokenizers python package is not installed. Please install it with `pip install tokenizers`

What it means

After onnxruntime loads, ONNXMiniLM_L6_V2.__init__ imports tokenizers and grabs tokenizers.Tokenizer; an ImportError is converted to ValueError with the pip hint. The HuggingFace tokenizers package provides the Rust fast tokenizer used to turn raw text into input_ids for the MiniLM ONNX model, so without it the EF cannot preprocess text at all.

Source

Thrown at chromadb/utils/embedding_functions/onnx_mini_lm_l6_v2.py:79

        if preferred_providers and len(preferred_providers) != len(
            set(preferred_providers)
        ):
            raise ValueError("Preferred providers must be unique")

        self._preferred_providers = preferred_providers

        try:
            # Equivalent to import onnxruntime
            self.ort = importlib.import_module("onnxruntime")
        except ImportError:
            raise ValueError(
                "The onnxruntime python package is not installed. Please install it with `pip install onnxruntime`"
            )
        try:
            # Equivalent to from tokenizers import Tokenizer
            self.Tokenizer = importlib.import_module("tokenizers").Tokenizer
        except ImportError:
            raise ValueError(
                "The tokenizers python package is not installed. Please install it with `pip install tokenizers`"
            )
        try:
            # Equivalent to from tqdm import tqdm
            self.tqdm = importlib.import_module("tqdm").tqdm
        except ImportError:
            raise ValueError(
                "The tqdm python package is not installed. Please install it with `pip install tqdm`"
            )

    # Borrowed from https://gist.github.com/yanqd0/c13ed29e29432e3cf3e7c38467f42f51
    # Download with tqdm to preserve the sentence-transformers experience
    @retry(  # type: ignore
        reraise=True,
        stop=stop_after_attempt(3),
        wait=wait_random(min=1, max=3),
        retry=retry_if_exception(lambda e: "does not match expected SHA256" in str(e)),
    )

View on GitHub (pinned to aecdd12c8a)

Solutions

pip install tokenizers
Reinstall chromadb cleanly to restore its full default dependency set: pip install --force-reinstall chromadb
Pin tokenizers in requirements alongside chromadb so image builds fail loudly when it is missing

Example fix

// before
fn = ONNXMiniLM_L6_V2()  # ValueError: tokenizers not installed

// after (shell)
pip install tokenizers
// then
fn = ONNXMiniLM_L6_V2()

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util
missing = [m for m in ("onnxruntime", "tokenizers", "tqdm") if importlib.util.find_spec(m) is None]
if missing:
    raise SystemExit(f"Missing packages for default EF: {missing}; pip install {' '.join(missing)}")

Try / catch

try:
    ef = ONNXMiniLM_L6_V2()
except ValueError as e:
    if "tokenizers" in str(e):
        raise SystemExit("Run: pip install tokenizers") from e
    raise

Prevention

Install the full chromadb default dependency set rather than cherry-picking wheels
Smoke-test EF construction in CI to catch stripped dependencies
Avoid mixing conda and pip for this dependency chain

When it happens

Trigger: Constructing ONNXMiniLM_L6_V2() in an environment where chromadb was installed without its default dependencies (e.g. from a minimal wheel or with --no-deps); tokenizers uninstalled by another package's resolver; partial/failed pip install that left onnxruntime present but tokenizers missing.

Common situations: Slim Docker images; mixing conda and pip where conda removed the pip-installed tokenizers; dependency conflicts where another library pins tokenizers to a version that failed to build and pip rolled it back.

Related errors

AI-assisted analysis of chroma-core/chroma@aecdd12c8a (2026-08-16). Data as JSON: /api/errors/d753c79c6d35b908. Report an issue: GitHub.