crewAIInc/crewAI · error · ValueError

Website URL must be provided.

Error message

Website URL must be provided.

What it means

Raised by ScrapeWebsiteTool._run when website_url resolves to None — kwargs.get('website_url', self.website_url) found neither a call-time argument nor an instance attribute. It fails before any HTTP request, and unlike the element scraper this tool has no other required parameters.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrape_website_tool/scrape_website_tool.py:75

            )

        if website_url is not None:
            self.website_url = website_url
            self.description = (
                f"A tool that can be used to read {website_url}'s content."
            )
            self.args_schema = FixedScrapeWebsiteToolSchema
            self._generate_description()
            if cookies is not None:
                self.cookies = {cookies["name"]: os.getenv(cookies["value"]) or ""}

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        website_url: str | None = kwargs.get("website_url", self.website_url)
        if website_url is None:
            raise ValueError("Website URL must be provided.")

        page = safe_get(
            website_url,
            timeout=15,
            headers=self.headers,
            cookies=self.cookies if self.cookies else {},
        )

        page.encoding = page.apparent_encoding
        parsed = BeautifulSoup(page.text, "html.parser")

        text = "The following text is scraped website content:\n\n"
        text += parsed.get_text(" ")
        text = re.sub("[ \t]+", " ", text)
        return re.sub("\\s+\n\\s+", "\n", text)

View on GitHub (pinned to 754d7323be)

Solutions

1. Pass website_url at run time: tool.run(website_url='https://example.com')
2. Or fix it at construction: ScrapeWebsiteTool(website_url='https://example.com')
3. Confirm the kwarg name is exactly website_url per FixedScrapeWebsiteToolSchema
4. Make the URL field required in the agent-facing tool schema so the LLM always supplies it

Example fix

# before
tool = ScrapeWebsiteTool()
tool.run()  # ValueError: Website URL must be provided.

# after
tool.run(website_url='https://example.com')

from urllib.parse import urlparse

def valid_target(url: str | None) -> bool:
    return isinstance(url, str) and bool(urlparse(url).scheme)

assert valid_target(target_url), "website_url (with scheme) is required"

try:
    tool.run(website_url=url)
except ValueError as e:
    if "must be provided" in str(e):
        raise ValueError("website_url missing — pass it to ScrapeWebsiteTool(...) or run(website_url=...)") from e
    raise

When it happens

Trigger: ScrapeWebsiteTool() constructed bare and run without a website_url kwarg; or calling with a misspelled/differently named key (e.g. 'url') so kwargs.get returns None and no default was set.

Common situations: Agents omitting the URL argument; tools constructed without the URL and expected to be parameterized per-call; version changes to the expected argument name; schema drift between the tool description and the LLM's emitted arguments.

Related errors

• Both website_url and css_element must be provided.
• Invalid URL scheme: {self.url}
• URL scheme must be 'http' or 'https'
• url is required either in constructor or method call
• url is required either in constructor or method call

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/9e32bc539271cf14. Report an issue: GitHub.