{"record":{"id":"21fc82c878391ded","repo":"crewAIInc/crewAI","slug":"beautifulsoup4-is-not-installed-please-install-it-21fc82","errorCode":null,"errorMessage":"beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`","messagePattern":"beautifulsoup4 is not installed\\. Please install it with `pip install crewai-tools\\[beautifulsoup4\\]`","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/scrape_website_tool/scrape_website_tool.py","lineNumber":55,"sourceCode":"        default_factory=lambda: {\n            \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36\",\n            \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\",\n            \"Accept-Language\": \"en-US,en;q=0.9\",\n            \"Referer\": \"https://www.google.com/\",\n            \"Connection\": \"keep-alive\",\n            \"Upgrade-Insecure-Requests\": \"1\",\n        }\n    )\n\n    def __init__(\n        self,\n        website_url: str | None = None,\n        cookies: dict[str, str] | None = None,\n        **kwargs: Any,\n    ):\n        super().__init__(**kwargs)\n        if not BEAUTIFULSOUP_AVAILABLE:\n            raise ImportError(\n                \"beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`\"\n            )\n\n        if website_url is not None:\n            self.website_url = website_url\n            self.description = (\n                f\"A tool that can be used to read {website_url}'s content.\"\n            )\n            self.args_schema = FixedScrapeWebsiteToolSchema\n            self._generate_description()\n            if cookies is not None:\n                self.cookies = {cookies[\"name\"]: os.getenv(cookies[\"value\"]) or \"\"}\n\n    def _run(\n        self,\n        **kwargs: Any,\n    ) -> Any:\n        website_url: str | None = kwargs.get(\"website_url\", self.website_url)","sourceCodeStart":37,"sourceCodeEnd":73,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/scrape_website_tool/scrape_website_tool.py#L37-L73","documentation":"Raised in ScrapeWebsiteTool.__init__ when BeautifulSoup4 is not importable. Unlike scrape_element_from_website (which defers to _run), this tool fails fast at construction: super().__init__ runs, then BEAUTIFULSOUP_AVAILABLE is checked and the ImportError is thrown immediately, so the error appears as soon as you build the tool.","triggerScenarios":"ScrapeWebsiteTool(...) constructed in any environment lacking beautifulsoup4 — i.e. crewai-tools installed without the [beautifulsoup4] extra (bs4 is optional, not a hard dependency).","commonSituations":"Fresh venv with plain crewai-tools; Docker images that pip-install the base package only; dependency lists regenerated without extras; teammates whose environments work because they happen to have bs4 from another package.","solutions":["Install the extra: pip install 'crewai-tools[beautifulsoup4]'","Or add beautifulsoup4 directly to the environment","Add the extra to pyproject/requirements so environments are reproducible","If using uv: uv add 'crewai-tools[beautifulsoup4]'"],"exampleFix":"# before\nfrom crewai_tools.tools.scrape_website_tool import ScrapeWebsiteTool\ntool = ScrapeWebsiteTool()  # ImportError at construction\n\n# after (install once)\n# pip install 'crewai-tools[beautifulsoup4]'\ntool = ScrapeWebsiteTool(website_url='https://example.com')\n","handlingStrategy":"validation","validationCode":"import importlib.util\n\ndef bs4_available() -> bool:\n    return importlib.util.find_spec(\"bs4\") is not None\n\nif not bs4_available():\n    raise SystemExit(\"Install first: pip install 'crewai-tools[beautifulsoup4]'\")","typeGuard":null,"tryCatchPattern":"try:\n    tool = ScrapeWebsiteTool(website_url=url)\nexcept ImportError as e:\n    raise SystemExit(f\"Missing dependency: {e}\") from e","preventionTips":["Declare crewai-tools[beautifulsoup4] (not bare crewai-tools) wherever dependencies are listed","Fail fast on find_spec('bs4') at app startup","Check container images include the extra","This tool fails at construction, so catching early avoids half-initialized agent setups"],"tags":["installation","dependencies","beautifulsoup","scraper","optional-extra"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}