crewAIInc/crewAI · error · ImportError

Reading HTML URLs requires beautifulsoup4. Install with: uv

Error message

Reading HTML URLs requires beautifulsoup4. Install with: uv add beautifulsoup4

What it means

UrlReadTool's HTML extraction lazily imports BeautifulSoup from the optional 'beautifulsoup4' package inside _extract_html. Reading any HTML URL without beautifulsoup4 installed raises ImportError with the install command.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/url_read_tool/url_read_tool.py:291

        except ImportError as e:
            raise ImportError(
                "Reading DOCX URLs requires python-docx. Install with: "
                "uv add python-docx"
            ) from e

        document = Document(BytesIO(body))
        return "\n".join(
            paragraph.text
            for paragraph in document.paragraphs
            if paragraph.text.strip()
        )

    def _extract_html(self, body: bytes, content_type: str) -> str:
        """Strip HTML bytes down to visible text."""
        try:
            from bs4 import BeautifulSoup
        except ImportError as e:
            raise ImportError(
                "Reading HTML URLs requires beautifulsoup4. Install with: "
                "uv add beautifulsoup4"
            ) from e

        soup = BeautifulSoup(self._decode(body, content_type), "html.parser")
        for element in soup(["script", "style"]):
            element.decompose()

        text = _SPACES_PATTERN.sub(" ", soup.get_text(" "))
        return _NEWLINE_PATTERN.sub("\n", text).strip()

    def _extract(self, body: bytes, kind: str, content_type: str) -> str:
        """Dispatch to the extractor named by *kind*."""
        if kind == "pdf":
            return self._extract_pdf(body)
        if kind == "docx":
            return self._extract_docx(body)
        if kind == "html":

View on GitHub (pinned to 754d7323be)

Solutions

Install the dependency: uv add beautifulsoup4 (or pip install beautifulsoup4).
Consider installing crewai-tools with its bundled extras so scraping deps come along.
Smoke-test UrlReadTool on one HTML URL at deploy time to catch missing parsers early.

Example fix

# before
tool = UrlReadTool()
tool.run('https://example.com')  # ImportError: requires beautifulsoup4

# after
# shell: uv add beautifulsoup4
tool.run('https://example.com')

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util

if not importlib.util.find_spec('bs4'):
    raise SystemExit('UrlReadTool needs beautifulsoup4 for HTML URLs: uv add beautifulsoup4')

Try / catch

try:
    text = tool.run(url)
except ImportError as e:
    if 'beautifulsoup4' in str(e):
        raise SystemExit('Install beautifulsoup4 to read HTML pages')
    raise

Prevention

Always install beautifulsoup4 alongside UrlReadTool - HTML is the default path.
Run a one-line HTML read smoke test in CI/deploy checks.
Keep a startup manifest of optional parsers required by your toolset.

When it happens

Trigger: Calling UrlReadTool on a text/html URL when bs4 is not installed; this is the default code path for most web pages, so virtually any web read triggers it in a bare environment.

Common situations: crewai-tools installed without its HTML extra, so the first web scrape fails; slim Docker images that pruned 'unused' packages; virtualenv recreated from a partial requirements list.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/8d4b442ff434e102. Report an issue: GitHub.