crewAIInc/crewAI · error · ValueError

Invalid URL format: {url}

Error message

Invalid URL format: {url}

What it means

SpiderTool validates the target URL through _validate_url before use and raises ValueError(f"Invalid URL format: {url}") when validation fails. Per the docstring, validation enforces a properly formatted HTTP(S) URL plus security constraints (e.g. scheme and network-location checks), so URLs that are malformed, non-HTTP(S), or unsafe are rejected.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py:172

                        and log_failures is True.

        Raises:
            ValueError: If URL is invalid or missing, or if mode is invalid.
            ImportError: If spider-client package is not properly installed.
            ConnectionError: If network connection fails while accessing the URL.
            Exception: For other runtime errors.
        """
        try:
            params = {}
            url = website_url or self.website_url

            if not url:
                raise ValueError(
                    "Website URL must be provided either during initialization or execution"
                )

            if not self._validate_url(url):
                raise ValueError(f"Invalid URL format: {url}")

            if mode not in ["scrape", "crawl"]:
                raise ValueError(
                    f"Invalid mode: {mode}. Must be either 'scrape' or 'crawl'"
                )

            params = {
                "request": self.config.DEFAULT_REQUEST_MODE,
                "filter_output_svg": self.config.FILTER_SVG,
                "return_format": self.config.DEFAULT_RETURN_FORMAT,
            }

            if mode == "crawl":
                params["limit"] = self.config.DEFAULT_CRAWL_LIMIT

            if self.custom_params:
                params.update(self.custom_params)

View on GitHub (pinned to 754d7323be)

Solutions

1. Normalize the URL before calling: strip whitespace and prepend https:// when the scheme is missing.
2. Use only http:// or https:// URLs with a valid hostname.
3. If you control the input pipeline, validate with urllib.parse first (scheme in {http,https} and netloc non-empty) and reject early.

Example fix

# before
result = tool._run(website_url="example.com/docs")

# after
from urllib.parse import urlparse
url = "example.com/docs" if "example.com/docs".startswith("http") else "https://example.com/docs"
assert urlparse(url).scheme in {"http", "https"} and urlparse(url).netloc
result = tool._run(website_url=url)

from urllib.parse import urlparse
url = url.strip()
if not url.startswith(("http://", "https://")):
    url = "https://" + url
p = urlparse(url)
if p.scheme not in {"http", "https"} or not p.netloc:
    raise ValueError(f"Refusing invalid URL: {url!r}")

def is_valid_http_url(url: str) -> bool:
    try:
        p = urlparse(url.strip())
    except ValueError:
        return False
    return p.scheme in {"http", "https"} and bool(p.netloc)

try:
    result = tool._run(website_url=url, mode=mode)
except ValueError as e:
    if "Invalid URL format" in str(e):
        url = "https://" + url.lstrip()
        result = tool._run(website_url=url, mode=mode)  # retry once, normalized
    else:
        raise

When it happens

Trigger: Passing 'example.com' (no scheme), 'ftp://example.com', 'javascript:...' or other non-http(s) schemes; URLs with spaces or invalid characters; SSRF-guarded targets (e.g. localhost/internal IPs) rejected by the security constraints; LLM-generated hallucinated URL strings.

Common situations: Agent-generated URLs lacking the https:// prefix; user input pasted without a scheme; attempts to scrape intranet/localhost addresses blocked by design; trailing whitespace breaking parsing.

Related errors

• URL scheme '{parsed.scheme}' is not allowed. Only http and h
• URL has no hostname: '{url}'
• file:// URLs are not allowed: '{url}'. Use a file path inste
• URL '{url}' resolves to private/reserved IP {ip_str}. Access
• Blocked unsafe {label}: {e}

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/2f124f03f03f2018. Report an issue: GitHub.