crewAIInc/crewAI · error · ValueError

Invalid URL format. URL must include scheme (http/https) and

Error message

Invalid URL format. URL must include scheme (http/https) and domain

What it means

Raised by the Pydantic field_validator on ScrapegraphScrapeToolInput.website_url when the URL cannot be parsed into a scheme plus netloc. urlparse is applied and both components must be truthy; any parse failure or missing part is re-raised as this fixed-message ValueError with the original error chained.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py:42

    """Input for ScrapegraphScrapeTool."""

    website_url: str = Field(..., description="Mandatory website url to scrape")
    user_prompt: str = Field(
        default="Extract the main content of the webpage",
        description="Prompt to guide the extraction of content",
    )

    @field_validator("website_url")
    @classmethod
    def validate_url(cls, v: str) -> str:
        """Validate URL format."""
        try:
            result = urlparse(v)
            if not all([result.scheme, result.netloc]):
                raise ValueError
            return v
        except Exception as e:
            raise ValueError(
                "Invalid URL format. URL must include scheme (http/https) and domain"
            ) from e


class ScrapegraphScrapeTool(BaseTool):
    """A tool that uses Scrapegraph AI to intelligently scrape website content.

    Raises:
        ValueError: If API key is missing or URL format is invalid
        RateLimitError: If API rate limits are exceeded
        RuntimeError: If scraping operation fails
    """

    model_config = ConfigDict(arbitrary_types_allowed=True)

    name: str = "Scrapegraph website scraper"
    description: str = (
        "A tool that uses Scrapegraph AI to intelligently scrape website content."

View on GitHub (pinned to 754d7323be)

Solutions

1. Prefix the scheme: use 'https://example.com/page' not 'example.com/page'
2. Strip whitespace/newlines from agent-supplied URLs before passing them in
3. If accepting user input, normalize with a preprocessing step that adds https:// when a scheme is missing
4. Validate URLs client-side before constructing the tool input

Example fix

# before
tool.run(website_url='docs.example.com/guide')  # ValueError: Invalid URL format

# after
tool.run(website_url='https://docs.example.com/guide')

from urllib.parse import urlparse

def is_full_url(u: str) -> bool:
    try:
        p = urlparse(u.strip())
    except ValueError:
        return False
    return bool(p.scheme) and bool(p.netloc)

assert is_full_url(website_url), "URL must include scheme and domain, e.g. https://example.com"

from typing import TypeGuard

def is_valid_website_url(v: str) -> TypeGuard[str]:
    p = urlparse(v.strip())
    return bool(p.scheme) and bool(p.netloc)

try:
    tool.run(website_url=u)
except ValueError as e:
    if "Invalid URL format" in str(e):
        u = u if "://" in u else "https://" + u
        tool.run(website_url=u)
    else:
        raise

When it happens

Trigger: Passing website_url values like 'example.com/page' (no scheme), 'https:///path' (no domain), 'ftp://x' is technically accepted if netloc exists, but strings lacking scheme or netloc — including garbage like 'not a url' — trigger validation failure at input-model construction.

Common situations: LLM agents returning bare domains without https://; copy-pasted URLs that lost the scheme; trailing-userinfo or whitespace-corrupted URLs failing urlparse; forgetting the protocol when hardcoding a URL in code.

Related errors

• Website URL cannot be empty
• Missing required input '{name}'{suffix}
• Invalid input '{location}': {error.get('msg')}
• Expected an async readable object with async read() method
• Expected a binary file-like object with read() and seek()

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/aa8a29e9ecf16f31. Report an issue: GitHub.