crewAIInc/crewAI · error · ValueError

Website URL must be provided either during initialization or

Error message

Website URL must be provided either during initialization or execution

What it means

SpiderTool._run requires a target website: it resolves website_url from the _run argument, falling back to self.website_url set at construction. If both are empty/None it raises ValueError telling you the URL must be supplied at initialization or execution. All of the tool's modes (scrape/crawl) operate on a URL, so there is no meaningful default.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py:167

                - "scrape": Extract content from single page
                - "crawl": Follow links and extract content from multiple pages

        Returns:
            Optional[str]: Extracted content in markdown format, or None if extraction fails
                        and log_failures is True.

        Raises:
            ValueError: If URL is invalid or missing, or if mode is invalid.
            ImportError: If spider-client package is not properly installed.
            ConnectionError: If network connection fails while accessing the URL.
            Exception: For other runtime errors.
        """
        try:
            params = {}
            url = website_url or self.website_url

            if not url:
                raise ValueError(
                    "Website URL must be provided either during initialization or execution"
                )

            if not self._validate_url(url):
                raise ValueError(f"Invalid URL format: {url}")

            if mode not in ["scrape", "crawl"]:
                raise ValueError(
                    f"Invalid mode: {mode}. Must be either 'scrape' or 'crawl'"
                )

            params = {
                "request": self.config.DEFAULT_REQUEST_MODE,
                "filter_output_svg": self.config.FILTER_SVG,
                "return_format": self.config.DEFAULT_RETURN_FORMAT,
            }

            if mode == "crawl":

View on GitHub (pinned to 754d7323be)

Solutions

Pass the URL at call time: tool._run(website_url="https://example.com", mode="scrape").
Or fix it for all calls: SpiderTool(website_url="https://example.com").
For agent usage, ensure the tool input schema marks the URL required (or set a default) so the model always supplies it.

Example fix

# before
tool = SpiderTool(api_key=...)
result = tool._run()

# after
result = tool._run(website_url="https://example.com", mode="scrape")

Defensive patterns

Strategy: validation

Validate before calling

url = (website_url or default_url or "").strip()
if not url:
    raise ValueError("website_url is required: pass it to SpiderTool(...) or _run(website_url=...)")
result = tool._run(website_url=url, mode=mode)

Type guard

def has_spider_target(tool_url: str | None, call_url: str | None) -> bool:
    return bool((call_url or tool_url or "").strip())

Try / catch

try:
    result = tool._run(website_url=url, mode=mode)
except ValueError as e:
    if "Website URL must be provided" in str(e):
        return "No URL supplied; provide a website_url."  # agent re-ask
    raise

Prevention

Set website_url at construction for single-site tools, or enforce a required url field in agent schemas.
Strip and check the URL string before invoking the tool.

When it happens

Trigger: Creating SpiderTool() without website_url and calling _run() with no website_url argument; passing website_url="" or None both places; an agent tool-call that omits the url parameter when the tool had no default.

Common situations: Agents expected to pass the URL at runtime but generating calls without it; configuration loading where the URL env var/setting is empty; reusable tool instances shared across tasks with no fixed target.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/c5c9992187fbbf19. Report an issue: GitHub.