unclecode/crawl4ai · error · ImportError

sentence-transformers is required for local embeddings. Inst

Error message

sentence-transformers is required for local embeddings. Install it with: pip install 'crawl4ai[transformer]' or pip install sentence-transformers

What it means

get_text_embeddings (crawl4ai/utils.py:3569) raises ImportError when the local-embedding branch is selected (no external embedding API configured) and `from sentence_transformers import SentenceTransformer` fails. The sentence-transformers stack is an optional heavy dependency (torch et al.) shipped only via the crawl4ai[transformer] extra, so default installs cannot do local embeddings.

Source

Thrown at crawl4ai/utils.py:3569

            kwargs['model'] = f"openai/{embedding_model}"
        
        # Get embeddings
        response = await aembedding(**kwargs)
        
        # Extract embeddings from response
        embeddings = []
        for item in response.data:
            embeddings.append(item['embedding'])
            
        return np.array(embeddings)
    
    # Default: use sentence-transformers
    else:
        # Lazy load to avoid importing heavy libraries unless needed
        try:
            from sentence_transformers import SentenceTransformer
        except ImportError:
            raise ImportError(
                "sentence-transformers is required for local embeddings. "
                "Install it with: pip install 'crawl4ai[transformer]' or pip install sentence-transformers"
            )
        
        # Cache the model in function attribute to avoid reloading
        if not hasattr(get_text_embeddings, '_models'):
            get_text_embeddings._models = {}
        
        if model_name not in get_text_embeddings._models:
            get_text_embeddings._models[model_name] = SentenceTransformer(model_name)
        
        encoder = get_text_embeddings._models[model_name]
        
        # Batch encode for efficiency
        embeddings = encoder.encode(
            texts,
            batch_size=batch_size,
            show_progress_bar=False,

View on GitHub (pinned to 7e80152142)

Solutions

pip install 'crawl4ai[transformer]' (or pip install sentence-transformers) in the same environment/venv as crawl4ai.
Alternatively configure an embedding API provider (the branch above the local one) so sentence-transformers is never imported.
In Docker, extend the image and add the extra, then rebuild.
Verify with python -c "from sentence_transformers import SentenceTransformer" to confirm the install is healthy.

Example fix

# before
embs = get_text_embeddings(texts)  # ImportError

# after (shell)
# pip install 'crawl4ai[transformer]'
embs = get_text_embeddings(texts)

Defensive patterns

Strategy: validation

Validate before calling

def local_embeddings_available() -> bool:
    try:
        from sentence_transformers import SentenceTransformer  # noqa
        return True
    except ImportError:
        return False

# if not local_embeddings_available(): configure API embedding provider instead

Try / catch

try:
    embs = get_text_embeddings(texts)
except ImportError as e:
    if "sentence-transformers" in str(e):
        raise SystemExit("Install: pip install 'crawl4ai[transformer]'")
    raise

Prevention

Install the transformer extra in the deployment image when local embeddings are used
Prefer the API-provider embedding branch in slim containers
Smoke-test the import in CI to catch trimmed dependencies

When it happens

Trigger: Calling get_text_embeddings(texts) with default/provider='local' semantics on a pip install crawl4ai without extras; running in a slim Docker image where torch/sentence-transformers were trimmed; a broken torch install causing the inner import to raise ImportError.

Common situations: Following docs that demo semantic chunking/embedding on a base install; deploying the Docker server image which excludes transformer extras to stay slim; Python version mismatch making sentence-transformers uninstallable.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/3156d4d3b115182f. Report an issue: GitHub.