{"record":{"id":"3156d4d3b115182f","repo":"unclecode/crawl4ai","slug":"sentence-transformers-is-required-for-local-embedd","errorCode":null,"errorMessage":"sentence-transformers is required for local embeddings. Install it with: pip install 'crawl4ai[transformer]' or pip install sentence-transformers","messagePattern":"sentence-transformers is required for local embeddings\\. Install it with: pip install 'crawl4ai\\[transformer\\]' or pip install sentence-transformers","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"crawl4ai/utils.py","lineNumber":3569,"sourceCode":"            kwargs['model'] = f\"openai/{embedding_model}\"\n        \n        # Get embeddings\n        response = await aembedding(**kwargs)\n        \n        # Extract embeddings from response\n        embeddings = []\n        for item in response.data:\n            embeddings.append(item['embedding'])\n            \n        return np.array(embeddings)\n    \n    # Default: use sentence-transformers\n    else:\n        # Lazy load to avoid importing heavy libraries unless needed\n        try:\n            from sentence_transformers import SentenceTransformer\n        except ImportError:\n            raise ImportError(\n                \"sentence-transformers is required for local embeddings. \"\n                \"Install it with: pip install 'crawl4ai[transformer]' or pip install sentence-transformers\"\n            )\n        \n        # Cache the model in function attribute to avoid reloading\n        if not hasattr(get_text_embeddings, '_models'):\n            get_text_embeddings._models = {}\n        \n        if model_name not in get_text_embeddings._models:\n            get_text_embeddings._models[model_name] = SentenceTransformer(model_name)\n        \n        encoder = get_text_embeddings._models[model_name]\n        \n        # Batch encode for efficiency\n        embeddings = encoder.encode(\n            texts,\n            batch_size=batch_size,\n            show_progress_bar=False,","sourceCodeStart":3551,"sourceCodeEnd":3587,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/utils.py#L3551-L3587","documentation":"get_text_embeddings (crawl4ai/utils.py:3569) raises ImportError when the local-embedding branch is selected (no external embedding API configured) and `from sentence_transformers import SentenceTransformer` fails. The sentence-transformers stack is an optional heavy dependency (torch et al.) shipped only via the crawl4ai[transformer] extra, so default installs cannot do local embeddings.","triggerScenarios":"Calling get_text_embeddings(texts) with default/provider='local' semantics on a pip install crawl4ai without extras; running in a slim Docker image where torch/sentence-transformers were trimmed; a broken torch install causing the inner import to raise ImportError.","commonSituations":"Following docs that demo semantic chunking/embedding on a base install; deploying the Docker server image which excludes transformer extras to stay slim; Python version mismatch making sentence-transformers uninstallable.","solutions":["pip install 'crawl4ai[transformer]' (or pip install sentence-transformers) in the same environment/venv as crawl4ai.","Alternatively configure an embedding API provider (the branch above the local one) so sentence-transformers is never imported.","In Docker, extend the image and add the extra, then rebuild.","Verify with python -c \"from sentence_transformers import SentenceTransformer\" to confirm the install is healthy."],"exampleFix":"# before\nembs = get_text_embeddings(texts)  # ImportError\n\n# after (shell)\n# pip install 'crawl4ai[transformer]'\nembs = get_text_embeddings(texts)","handlingStrategy":"validation","validationCode":"def local_embeddings_available() -> bool:\n    try:\n        from sentence_transformers import SentenceTransformer  # noqa\n        return True\n    except ImportError:\n        return False\n\n# if not local_embeddings_available(): configure API embedding provider instead","typeGuard":null,"tryCatchPattern":"try:\n    embs = get_text_embeddings(texts)\nexcept ImportError as e:\n    if \"sentence-transformers\" in str(e):\n        raise SystemExit(\"Install: pip install 'crawl4ai[transformer]'\")\n    raise","preventionTips":["Install the transformer extra in the deployment image when local embeddings are used","Prefer the API-provider embedding branch in slim containers","Smoke-test the import in CI to catch trimmed dependencies"],"tags":["dependency","embeddings","sentence-transformers","crawl4ai"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}