{"record":{"id":"8d4b442ff434e102","repo":"crewAIInc/crewAI","slug":"reading-html-urls-requires-beautifulsoup4-install","errorCode":null,"errorMessage":"Reading HTML URLs requires beautifulsoup4. Install with: uv add beautifulsoup4","messagePattern":"Reading HTML URLs requires beautifulsoup4\\. Install with: uv add beautifulsoup4","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/url_read_tool/url_read_tool.py","lineNumber":291,"sourceCode":"        except ImportError as e:\n            raise ImportError(\n                \"Reading DOCX URLs requires python-docx. Install with: \"\n                \"uv add python-docx\"\n            ) from e\n\n        document = Document(BytesIO(body))\n        return \"\\n\".join(\n            paragraph.text\n            for paragraph in document.paragraphs\n            if paragraph.text.strip()\n        )\n\n    def _extract_html(self, body: bytes, content_type: str) -> str:\n        \"\"\"Strip HTML bytes down to visible text.\"\"\"\n        try:\n            from bs4 import BeautifulSoup\n        except ImportError as e:\n            raise ImportError(\n                \"Reading HTML URLs requires beautifulsoup4. Install with: \"\n                \"uv add beautifulsoup4\"\n            ) from e\n\n        soup = BeautifulSoup(self._decode(body, content_type), \"html.parser\")\n        for element in soup([\"script\", \"style\"]):\n            element.decompose()\n\n        text = _SPACES_PATTERN.sub(\" \", soup.get_text(\" \"))\n        return _NEWLINE_PATTERN.sub(\"\\n\", text).strip()\n\n    def _extract(self, body: bytes, kind: str, content_type: str) -> str:\n        \"\"\"Dispatch to the extractor named by *kind*.\"\"\"\n        if kind == \"pdf\":\n            return self._extract_pdf(body)\n        if kind == \"docx\":\n            return self._extract_docx(body)\n        if kind == \"html\":","sourceCodeStart":273,"sourceCodeEnd":309,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/url_read_tool/url_read_tool.py#L273-L309","documentation":"UrlReadTool's HTML extraction lazily imports BeautifulSoup from the optional 'beautifulsoup4' package inside _extract_html. Reading any HTML URL without beautifulsoup4 installed raises ImportError with the install command.","triggerScenarios":"Calling UrlReadTool on a text/html URL when bs4 is not installed; this is the default code path for most web pages, so virtually any web read triggers it in a bare environment.","commonSituations":"crewai-tools installed without its HTML extra, so the first web scrape fails; slim Docker images that pruned 'unused' packages; virtualenv recreated from a partial requirements list.","solutions":["Install the dependency: uv add beautifulsoup4 (or pip install beautifulsoup4).","Consider installing crewai-tools with its bundled extras so scraping deps come along.","Smoke-test UrlReadTool on one HTML URL at deploy time to catch missing parsers early."],"exampleFix":"# before\ntool = UrlReadTool()\ntool.run('https://example.com')  # ImportError: requires beautifulsoup4\n\n# after\n# shell: uv add beautifulsoup4\ntool.run('https://example.com')","handlingStrategy":"validation","validationCode":"import importlib.util\n\nif not importlib.util.find_spec('bs4'):\n    raise SystemExit('UrlReadTool needs beautifulsoup4 for HTML URLs: uv add beautifulsoup4')","typeGuard":null,"tryCatchPattern":"try:\n    text = tool.run(url)\nexcept ImportError as e:\n    if 'beautifulsoup4' in str(e):\n        raise SystemExit('Install beautifulsoup4 to read HTML pages')\n    raise","preventionTips":["Always install beautifulsoup4 alongside UrlReadTool - HTML is the default path.","Run a one-line HTML read smoke test in CI/deploy checks.","Keep a startup manifest of optional parsers required by your toolset."],"tags":["dependencies","html","beautifulsoup4","scraping","url-read"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}