crewAIInc/crewAI · error · ImportError

beautifulsoup4 is not installed. Please install it with `pip

Error message

beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`

What it means

Raised at call time (_run) by ScrapeElementFromWebsiteTool when BeautifulSoup4 is not importable. The module guards its bs4 import with BEAUTIFULSOUP_AVAILABLE; unlike tools that fail at construction, this one lets you instantiate it and only fails when you actually run a scrape, so the error can surface late — e.g. inside an agent task.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrape_element_from_website/scrape_element_from_website.py:75

    ):
        super().__init__(**kwargs)
        if website_url is not None:
            self.website_url = website_url
            self.css_element = css_element
            self.description = (
                f"A tool that can be used to read {website_url}'s content."
            )
            self.args_schema = FixedScrapeElementFromWebsiteToolSchema
            self._generate_description()
            if cookies is not None:
                self.cookies = {cookies["name"]: os.getenv(cookies["value"]) or ""}

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        if not BEAUTIFULSOUP_AVAILABLE:
            raise ImportError(
                "beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`"
            )

        website_url = kwargs.get("website_url", self.website_url)
        css_element = kwargs.get("css_element", self.css_element)

        if website_url is None or css_element is None:
            raise ValueError("Both website_url and css_element must be provided.")

        page = safe_get(
            website_url,
            headers=self.headers,
            cookies=self.cookies if self.cookies else {},
            timeout=30,
        )
        parsed = BeautifulSoup(page.content, "html.parser")
        elements = parsed.select(css_element)
        return "\n".join([element.get_text() for element in elements])

View on GitHub (pinned to 754d7323be)

Solutions

Install the extra: pip install 'crewai-tools[beautifulsoup4]' (or uv add 'crewai-tools[beautifulsoup4]')
Alternatively install bs4 directly: pip install beautifulsoup4
Pin the extra in requirements/pyproject so rebuilt environments include it
Optionally check availability at construction time to fail fast before an agent run begins

Example fix

# before
python -m venv .venv && pip install crewai-tools
# tool.run(...) -> ImportError: beautifulsoup4 is not installed

# after
pip install 'crewai-tools[beautifulsoup4]'

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util

def bs4_available() -> bool:
    return importlib.util.find_spec("bs4") is not None

assert bs4_available(), "pip install 'crewai-tools[beautifulsoup4]'"

Try / catch

try:
    result = tool.run(website_url=u, css_element=sel)
except ImportError as e:
    raise SystemExit("beautifulsoup4 missing: pip install 'crewai-tools[beautifulsoup4]'") from e

Prevention

Install the [beautifulsoup4] extra in every environment definition (pyproject, requirements, Dockerfile)
Add a startup smoke test checking find_spec('bs4') so failure surfaces before agent runs
Because the check happens in _run, not __init__, verify at construction time yourself
Pin crewai-tools and its extras together to avoid partial installs

When it happens

Trigger: Constructing ScrapeElementFromWebsiteTool and calling _run/run in an environment where beautifulsoup4 was never installed — plain `pip install crewai-tools` does not pull it in; it is an optional extra.

Common situations: Minimal installs that skip extras; new venvs recreated from a trimmed requirements.txt; CI images missing the [beautifulsoup4] extra; the error appearing only mid-agent-run instead of at setup, which obscures the cause.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/bbda331708f4f62a. Report an issue: GitHub.