What causes "website_url is required" in crewAIInc/crewAI?

Pass website_url to the constructor: ScrapegraphScrapeTool(website_url='https://example.com', ...).

crewAIInc/crewAI · error · ValueError

website_url is required

Error message

website_url is required

What it means

Raised as ValueError by ScrapegraphScrapeTool._run when website_url resolves to a falsy value after checking both the _run kwargs and the tool instance attribute. The tool accepts the URL either as a constructor argument (website_url=...) or per-call kwarg; if neither is supplied, scraping cannot proceed. This is a pure usage error — the API is never contacted.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py:170

            raise RuntimeError(f"API error: {error_msg}")

        if "result" not in response:
            raise RuntimeError("Invalid response format from Scrapegraph API")

        return str(response["result"])

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        website_url = kwargs.get("website_url", self.website_url)
        user_prompt = (
            kwargs.get("user_prompt", self.user_prompt)
            or "Extract the main content of the webpage"
        )

        if not website_url:
            raise ValueError("website_url is required")

        self._validate_url(website_url)

        try:
            if self._client is None:
                raise RuntimeError("Client not initialized")
            return self._client.smartscraper(
                website_url=website_url,
                user_prompt=user_prompt,
            )

        except RateLimitError:
            raise  # Re-raise rate limit errors
        except Exception as e:
            raise RuntimeError(f"Scraping failed: {e!s}") from e
        finally:
            # Always close the client
            if self._client is not None:

View on GitHub (pinned to 754d7323be)

Solutions

Pass website_url to the constructor: ScrapegraphScrapeTool(website_url='https://example.com', ...).
Or pass it per call in the kwargs: tool.run(website_url='https://example.com').
Check for typos in the kwarg name — only 'website_url' is read via kwargs.get.

Example fix

# before
tool = ScrapegraphScrapeTool(api_key=KEY)
tool.run()  # ValueError: website_url is required

# after
tool = ScrapegraphScrapeTool(api_key=KEY)
tool.run(website_url="https://example.com")

Defensive patterns

Strategy: validation

Validate before calling

def ensure_url(url: str | None) -> str:
    if not url or not url.strip():
        raise ValueError("website_url is required")
    return url.strip()

url = ensure_url(maybe_url)
result = tool.run(website_url=url)

Try / catch

try:
    result = tool.run(website_url=url)
except ValueError as e:
    if "website_url is required" in str(e):
        url = prompt_or_default_url()  # recover
        result = tool.run(website_url=url)
    raise

Prevention

Always pass website_url — at construction or per call — never both empty.
Exact kwarg name 'website_url'; other names are silently ignored by kwargs.get.
Assert required URL config before building crews so failures happen at startup, not mid-run.

When it happens

Trigger: Instantiating ScrapegraphScrapeTool() with no website_url and calling _run(**kwargs) without a website_url kwarg; or explicitly passing website_url=None on both the constructor and the call.

Common situations: Building the tool dynamically (e.g. from config) where the URL field is optional and forgotten; passing the URL under a different kwarg name (typo like 'url' or 'web_url'); a template that constructs tools without arguments intending to pass URLs at run time but the agent omits the parameter.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/5864171b2b8d9533. Report an issue: GitHub.