{"record":{"id":"bbda331708f4f62a","repo":"crewAIInc/crewAI","slug":"beautifulsoup4-is-not-installed-please-install-it","errorCode":null,"errorMessage":"beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`","messagePattern":"beautifulsoup4 is not installed\\. Please install it with `pip install crewai-tools\\[beautifulsoup4\\]`","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/scrape_element_from_website/scrape_element_from_website.py","lineNumber":75,"sourceCode":"    ):\n        super().__init__(**kwargs)\n        if website_url is not None:\n            self.website_url = website_url\n            self.css_element = css_element\n            self.description = (\n                f\"A tool that can be used to read {website_url}'s content.\"\n            )\n            self.args_schema = FixedScrapeElementFromWebsiteToolSchema\n            self._generate_description()\n            if cookies is not None:\n                self.cookies = {cookies[\"name\"]: os.getenv(cookies[\"value\"]) or \"\"}\n\n    def _run(\n        self,\n        **kwargs: Any,\n    ) -> Any:\n        if not BEAUTIFULSOUP_AVAILABLE:\n            raise ImportError(\n                \"beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`\"\n            )\n\n        website_url = kwargs.get(\"website_url\", self.website_url)\n        css_element = kwargs.get(\"css_element\", self.css_element)\n\n        if website_url is None or css_element is None:\n            raise ValueError(\"Both website_url and css_element must be provided.\")\n\n        page = safe_get(\n            website_url,\n            headers=self.headers,\n            cookies=self.cookies if self.cookies else {},\n            timeout=30,\n        )\n        parsed = BeautifulSoup(page.content, \"html.parser\")\n        elements = parsed.select(css_element)\n        return \"\\n\".join([element.get_text() for element in elements])","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/scrape_element_from_website/scrape_element_from_website.py#L57-L93","documentation":"Raised at call time (_run) by ScrapeElementFromWebsiteTool when BeautifulSoup4 is not importable. The module guards its bs4 import with BEAUTIFULSOUP_AVAILABLE; unlike tools that fail at construction, this one lets you instantiate it and only fails when you actually run a scrape, so the error can surface late — e.g. inside an agent task.","triggerScenarios":"Constructing ScrapeElementFromWebsiteTool and calling _run/run in an environment where beautifulsoup4 was never installed — plain `pip install crewai-tools` does not pull it in; it is an optional extra.","commonSituations":"Minimal installs that skip extras; new venvs recreated from a trimmed requirements.txt; CI images missing the [beautifulsoup4] extra; the error appearing only mid-agent-run instead of at setup, which obscures the cause.","solutions":["Install the extra: pip install 'crewai-tools[beautifulsoup4]' (or uv add 'crewai-tools[beautifulsoup4]')","Alternatively install bs4 directly: pip install beautifulsoup4","Pin the extra in requirements/pyproject so rebuilt environments include it","Optionally check availability at construction time to fail fast before an agent run begins"],"exampleFix":"# before\npython -m venv .venv && pip install crewai-tools\n# tool.run(...) -> ImportError: beautifulsoup4 is not installed\n\n# after\npip install 'crewai-tools[beautifulsoup4]'\n","handlingStrategy":"validation","validationCode":"import importlib.util\n\ndef bs4_available() -> bool:\n    return importlib.util.find_spec(\"bs4\") is not None\n\nassert bs4_available(), \"pip install 'crewai-tools[beautifulsoup4]'\"","typeGuard":null,"tryCatchPattern":"try:\n    result = tool.run(website_url=u, css_element=sel)\nexcept ImportError as e:\n    raise SystemExit(\"beautifulsoup4 missing: pip install 'crewai-tools[beautifulsoup4]'\") from e","preventionTips":["Install the [beautifulsoup4] extra in every environment definition (pyproject, requirements, Dockerfile)","Add a startup smoke test checking find_spec('bs4') so failure surfaces before agent runs","Because the check happens in _run, not __init__, verify at construction time yourself","Pin crewai-tools and its extras together to avoid partial installs"],"tags":["installation","dependencies","beautifulsoup","scraper","optional-extra"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}