{"record":{"id":"2f124f03f03f2018","repo":"crewAIInc/crewAI","slug":"invalid-url-format-url","errorCode":null,"errorMessage":"Invalid URL format: {url}","messagePattern":"Invalid URL format: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py","lineNumber":172,"sourceCode":"                        and log_failures is True.\n\n        Raises:\n            ValueError: If URL is invalid or missing, or if mode is invalid.\n            ImportError: If spider-client package is not properly installed.\n            ConnectionError: If network connection fails while accessing the URL.\n            Exception: For other runtime errors.\n        \"\"\"\n        try:\n            params = {}\n            url = website_url or self.website_url\n\n            if not url:\n                raise ValueError(\n                    \"Website URL must be provided either during initialization or execution\"\n                )\n\n            if not self._validate_url(url):\n                raise ValueError(f\"Invalid URL format: {url}\")\n\n            if mode not in [\"scrape\", \"crawl\"]:\n                raise ValueError(\n                    f\"Invalid mode: {mode}. Must be either 'scrape' or 'crawl'\"\n                )\n\n            params = {\n                \"request\": self.config.DEFAULT_REQUEST_MODE,\n                \"filter_output_svg\": self.config.FILTER_SVG,\n                \"return_format\": self.config.DEFAULT_RETURN_FORMAT,\n            }\n\n            if mode == \"crawl\":\n                params[\"limit\"] = self.config.DEFAULT_CRAWL_LIMIT\n\n            if self.custom_params:\n                params.update(self.custom_params)\n","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py#L154-L190","documentation":"SpiderTool validates the target URL through _validate_url before use and raises ValueError(f\"Invalid URL format: {url}\") when validation fails. Per the docstring, validation enforces a properly formatted HTTP(S) URL plus security constraints (e.g. scheme and network-location checks), so URLs that are malformed, non-HTTP(S), or unsafe are rejected.","triggerScenarios":"Passing 'example.com' (no scheme), 'ftp://example.com', 'javascript:...' or other non-http(s) schemes; URLs with spaces or invalid characters; SSRF-guarded targets (e.g. localhost/internal IPs) rejected by the security constraints; LLM-generated hallucinated URL strings.","commonSituations":"Agent-generated URLs lacking the https:// prefix; user input pasted without a scheme; attempts to scrape intranet/localhost addresses blocked by design; trailing whitespace breaking parsing.","solutions":["Normalize the URL before calling: strip whitespace and prepend https:// when the scheme is missing.","Use only http:// or https:// URLs with a valid hostname.","If you control the input pipeline, validate with urllib.parse first (scheme in {http,https} and netloc non-empty) and reject early."],"exampleFix":"# before\nresult = tool._run(website_url=\"example.com/docs\")\n\n# after\nfrom urllib.parse import urlparse\nurl = \"example.com/docs\" if \"example.com/docs\".startswith(\"http\") else \"https://example.com/docs\"\nassert urlparse(url).scheme in {\"http\", \"https\"} and urlparse(url).netloc\nresult = tool._run(website_url=url)","handlingStrategy":"type-guard","validationCode":"from urllib.parse import urlparse\nurl = url.strip()\nif not url.startswith((\"http://\", \"https://\")):\n    url = \"https://\" + url\np = urlparse(url)\nif p.scheme not in {\"http\", \"https\"} or not p.netloc:\n    raise ValueError(f\"Refusing invalid URL: {url!r}\")","typeGuard":"def is_valid_http_url(url: str) -> bool:\n    try:\n        p = urlparse(url.strip())\n    except ValueError:\n        return False\n    return p.scheme in {\"http\", \"https\"} and bool(p.netloc)","tryCatchPattern":"try:\n    result = tool._run(website_url=url, mode=mode)\nexcept ValueError as e:\n    if \"Invalid URL format\" in str(e):\n        url = \"https://\" + url.lstrip()\n        result = tool._run(website_url=url, mode=mode)  # retry once, normalized\n    else:\n        raise","preventionTips":["Normalize and parse URLs with urllib.parse before passing them to the tool.","For agent inputs, validate the model-emitted URL with a type guard and repair a missing scheme."],"tags":["spider","url-validation","security","input-validation"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}