{"record":{"id":"d753c79c6d35b908","repo":"chroma-core/chroma","slug":"the-tokenizers-python-package-is-not-installed-pl","errorCode":null,"errorMessage":"The tokenizers python package is not installed. Please install it with `pip install tokenizers`","messagePattern":"The tokenizers python package is not installed\\. Please install it with `pip install tokenizers`","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"chromadb/utils/embedding_functions/onnx_mini_lm_l6_v2.py","lineNumber":79,"sourceCode":"        if preferred_providers and len(preferred_providers) != len(\n            set(preferred_providers)\n        ):\n            raise ValueError(\"Preferred providers must be unique\")\n\n        self._preferred_providers = preferred_providers\n\n        try:\n            # Equivalent to import onnxruntime\n            self.ort = importlib.import_module(\"onnxruntime\")\n        except ImportError:\n            raise ValueError(\n                \"The onnxruntime python package is not installed. Please install it with `pip install onnxruntime`\"\n            )\n        try:\n            # Equivalent to from tokenizers import Tokenizer\n            self.Tokenizer = importlib.import_module(\"tokenizers\").Tokenizer\n        except ImportError:\n            raise ValueError(\n                \"The tokenizers python package is not installed. Please install it with `pip install tokenizers`\"\n            )\n        try:\n            # Equivalent to from tqdm import tqdm\n            self.tqdm = importlib.import_module(\"tqdm\").tqdm\n        except ImportError:\n            raise ValueError(\n                \"The tqdm python package is not installed. Please install it with `pip install tqdm`\"\n            )\n\n    # Borrowed from https://gist.github.com/yanqd0/c13ed29e29432e3cf3e7c38467f42f51\n    # Download with tqdm to preserve the sentence-transformers experience\n    @retry(  # type: ignore\n        reraise=True,\n        stop=stop_after_attempt(3),\n        wait=wait_random(min=1, max=3),\n        retry=retry_if_exception(lambda e: \"does not match expected SHA256\" in str(e)),\n    )","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/chroma-core/chroma/blob/aecdd12c8a891610db8653630b066b32ceb678b5/chromadb/utils/embedding_functions/onnx_mini_lm_l6_v2.py#L61-L97","documentation":"After onnxruntime loads, ONNXMiniLM_L6_V2.__init__ imports tokenizers and grabs tokenizers.Tokenizer; an ImportError is converted to ValueError with the pip hint. The HuggingFace tokenizers package provides the Rust fast tokenizer used to turn raw text into input_ids for the MiniLM ONNX model, so without it the EF cannot preprocess text at all.","triggerScenarios":"Constructing ONNXMiniLM_L6_V2() in an environment where chromadb was installed without its default dependencies (e.g. from a minimal wheel or with --no-deps); tokenizers uninstalled by another package's resolver; partial/failed pip install that left onnxruntime present but tokenizers missing.","commonSituations":"Slim Docker images; mixing conda and pip where conda removed the pip-installed tokenizers; dependency conflicts where another library pins tokenizers to a version that failed to build and pip rolled it back.","solutions":["pip install tokenizers","Reinstall chromadb cleanly to restore its full default dependency set: pip install --force-reinstall chromadb","Pin tokenizers in requirements alongside chromadb so image builds fail loudly when it is missing"],"exampleFix":"// before\nfn = ONNXMiniLM_L6_V2()  # ValueError: tokenizers not installed\n\n// after (shell)\npip install tokenizers\n// then\nfn = ONNXMiniLM_L6_V2()","handlingStrategy":"validation","validationCode":"import importlib.util\nmissing = [m for m in (\"onnxruntime\", \"tokenizers\", \"tqdm\") if importlib.util.find_spec(m) is None]\nif missing:\n    raise SystemExit(f\"Missing packages for default EF: {missing}; pip install {' '.join(missing)}\")","typeGuard":null,"tryCatchPattern":"try:\n    ef = ONNXMiniLM_L6_V2()\nexcept ValueError as e:\n    if \"tokenizers\" in str(e):\n        raise SystemExit(\"Run: pip install tokenizers\") from e\n    raise","preventionTips":["Install the full chromadb default dependency set rather than cherry-picking wheels","Smoke-test EF construction in CI to catch stripped dependencies","Avoid mixing conda and pip for this dependency chain"],"tags":["onnx","embedding-function","tokenizers","import-error","dependency","chroma"],"backgroundTag":"missing-dependency","analyzedSha":"aecdd12c8a891610db8653630b066b32ceb678b5","analyzedAt":"2026-08-16T21:53:27.228Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}