{"record":{"id":"aa8a29e9ecf16f31","repo":"crewAIInc/crewAI","slug":"invalid-url-format-url-must-include-scheme-http","errorCode":null,"errorMessage":"Invalid URL format. URL must include scheme (http/https) and domain","messagePattern":"Invalid URL format\\. URL must include scheme \\(http/https\\) and domain","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py","lineNumber":42,"sourceCode":"    \"\"\"Input for ScrapegraphScrapeTool.\"\"\"\n\n    website_url: str = Field(..., description=\"Mandatory website url to scrape\")\n    user_prompt: str = Field(\n        default=\"Extract the main content of the webpage\",\n        description=\"Prompt to guide the extraction of content\",\n    )\n\n    @field_validator(\"website_url\")\n    @classmethod\n    def validate_url(cls, v: str) -> str:\n        \"\"\"Validate URL format.\"\"\"\n        try:\n            result = urlparse(v)\n            if not all([result.scheme, result.netloc]):\n                raise ValueError\n            return v\n        except Exception as e:\n            raise ValueError(\n                \"Invalid URL format. URL must include scheme (http/https) and domain\"\n            ) from e\n\n\nclass ScrapegraphScrapeTool(BaseTool):\n    \"\"\"A tool that uses Scrapegraph AI to intelligently scrape website content.\n\n    Raises:\n        ValueError: If API key is missing or URL format is invalid\n        RateLimitError: If API rate limits are exceeded\n        RuntimeError: If scraping operation fails\n    \"\"\"\n\n    model_config = ConfigDict(arbitrary_types_allowed=True)\n\n    name: str = \"Scrapegraph website scraper\"\n    description: str = (\n        \"A tool that uses Scrapegraph AI to intelligently scrape website content.\"","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/scrapegraph_scrape_tool/scrapegraph_scrape_tool.py#L24-L60","documentation":"Raised by the Pydantic field_validator on ScrapegraphScrapeToolInput.website_url when the URL cannot be parsed into a scheme plus netloc. urlparse is applied and both components must be truthy; any parse failure or missing part is re-raised as this fixed-message ValueError with the original error chained.","triggerScenarios":"Passing website_url values like 'example.com/page' (no scheme), 'https:///path' (no domain), 'ftp://x' is technically accepted if netloc exists, but strings lacking scheme or netloc — including garbage like 'not a url' — trigger validation failure at input-model construction.","commonSituations":"LLM agents returning bare domains without https://; copy-pasted URLs that lost the scheme; trailing-userinfo or whitespace-corrupted URLs failing urlparse; forgetting the protocol when hardcoding a URL in code.","solutions":["Prefix the scheme: use 'https://example.com/page' not 'example.com/page'","Strip whitespace/newlines from agent-supplied URLs before passing them in","If accepting user input, normalize with a preprocessing step that adds https:// when a scheme is missing","Validate URLs client-side before constructing the tool input"],"exampleFix":"# before\ntool.run(website_url='docs.example.com/guide')  # ValueError: Invalid URL format\n\n# after\ntool.run(website_url='https://docs.example.com/guide')\n","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\ndef is_full_url(u: str) -> bool:\n    try:\n        p = urlparse(u.strip())\n    except ValueError:\n        return False\n    return bool(p.scheme) and bool(p.netloc)\n\nassert is_full_url(website_url), \"URL must include scheme and domain, e.g. https://example.com\"","typeGuard":"from typing import TypeGuard\n\ndef is_valid_website_url(v: str) -> TypeGuard[str]:\n    p = urlparse(v.strip())\n    return bool(p.scheme) and bool(p.netloc)","tryCatchPattern":"try:\n    tool.run(website_url=u)\nexcept ValueError as e:\n    if \"Invalid URL format\" in str(e):\n        u = u if \"://\" in u else \"https://\" + u\n        tool.run(website_url=u)\n    else:\n        raise","preventionTips":["Always emit full https:// URLs from agents (state it in the tool description)","Strip whitespace from LLM-returned URLs before passing them in","Prepend https:// when a scheme is missing in user input","Validate scheme+netloc client-side with urlparse before constructing tool inputs"],"tags":["validation","url","pydantic","scrapegraph"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}