crewAIInc/crewAI · error · ImportError

beautifulsoup4 is not installed. Please install it with `pip

Error message

beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`

What it means

Raised in ScrapeWebsiteTool.__init__ when BeautifulSoup4 is not importable. Unlike scrape_element_from_website (which defers to _run), this tool fails fast at construction: super().__init__ runs, then BEAUTIFULSOUP_AVAILABLE is checked and the ImportError is thrown immediately, so the error appears as soon as you build the tool.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrape_website_tool/scrape_website_tool.py:55

        default_factory=lambda: {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
            "Accept-Language": "en-US,en;q=0.9",
            "Referer": "https://www.google.com/",
            "Connection": "keep-alive",
            "Upgrade-Insecure-Requests": "1",
        }
    )

    def __init__(
        self,
        website_url: str | None = None,
        cookies: dict[str, str] | None = None,
        **kwargs: Any,
    ):
        super().__init__(**kwargs)
        if not BEAUTIFULSOUP_AVAILABLE:
            raise ImportError(
                "beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`"
            )

        if website_url is not None:
            self.website_url = website_url
            self.description = (
                f"A tool that can be used to read {website_url}'s content."
            )
            self.args_schema = FixedScrapeWebsiteToolSchema
            self._generate_description()
            if cookies is not None:
                self.cookies = {cookies["name"]: os.getenv(cookies["value"]) or ""}

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        website_url: str | None = kwargs.get("website_url", self.website_url)

View on GitHub (pinned to 754d7323be)

Solutions

Install the extra: pip install 'crewai-tools[beautifulsoup4]'
Or add beautifulsoup4 directly to the environment
Add the extra to pyproject/requirements so environments are reproducible
If using uv: uv add 'crewai-tools[beautifulsoup4]'

Example fix

# before
from crewai_tools.tools.scrape_website_tool import ScrapeWebsiteTool
tool = ScrapeWebsiteTool()  # ImportError at construction

# after (install once)
# pip install 'crewai-tools[beautifulsoup4]'
tool = ScrapeWebsiteTool(website_url='https://example.com')

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util

def bs4_available() -> bool:
    return importlib.util.find_spec("bs4") is not None

if not bs4_available():
    raise SystemExit("Install first: pip install 'crewai-tools[beautifulsoup4]'")

Try / catch

try:
    tool = ScrapeWebsiteTool(website_url=url)
except ImportError as e:
    raise SystemExit(f"Missing dependency: {e}") from e

Prevention

Declare crewai-tools[beautifulsoup4] (not bare crewai-tools) wherever dependencies are listed
Fail fast on find_spec('bs4') at app startup
Check container images include the extra
This tool fails at construction, so catching early avoids half-initialized agent setups

When it happens

Trigger: ScrapeWebsiteTool(...) constructed in any environment lacking beautifulsoup4 — i.e. crewai-tools installed without the [beautifulsoup4] extra (bs4 is optional, not a hard dependency).

Common situations: Fresh venv with plain crewai-tools; Docker images that pip-install the base package only; dependency lists regenerated without extras; teammates whose environments work because they happen to have bs4 from another package.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/21fc82c878391ded. Report an issue: GitHub.