{"record":{"id":"3a5a502a3cb516a6","repo":"unclecode/crawl4ai","slug":"unsupported-url-scheme-scheme","errorCode":null,"errorMessage":"Unsupported URL scheme: {scheme}","messagePattern":"Unsupported URL scheme: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_crawler_strategy.py","lineNumber":2810,"sourceCode":" raise ConnectionTimeoutError(f\"Request timed out: {str(e)}\")\n \n except Exception as e:\n await self.hooks['on_error'](e)\n raise HTTPCrawlerError(f\"HTTP request failed: {str(e)}\")\n\n async def crawl(\n self, \n url: str, \n config: Optional[CrawlerRunConfig] = None, \n **kwargs\n ) -> AsyncCrawlResponse:\n config = config or CrawlerRunConfig.from_kwargs(kwargs)\n \n parsed = urlparse(url)\n scheme = parsed.scheme.rstrip('/')\n \n if scheme not in self.VALID_SCHEMES:\n raise ValueError(f\"Unsupported URL scheme: {scheme}\")\n \n try:\n if scheme == 'file':\n return await self._handle_file(parsed.path)\n elif scheme == 'raw':\n # Don't use parsed.path - urlparse truncates at '#' which is common in CSS\n # Strip prefix directly: \"raw://\" (6 chars) or \"raw:\" (4 chars)\n raw_content = url[6:] if url.startswith(\"raw://\") else url[4:]\n return await self._handle_raw(raw_content, base_url=config.base_url)\n else: # http or https\n return await self._handle_http(url, config)\n \n except Exception as e:\n if self.logger:\n self.logger.error(\n message=\"Crawl failed: {error}\",\n tag=\"CRAWL\",\n params={\"error\": str(e), \"url\": url}","sourceCodeStart":2792,"sourceCodeEnd":2828,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_crawler_strategy.py#L2792-L2828","documentation":"Entry-point validation of the HTTP-mode crawler: the URL's scheme (from urlparse, trailing '/' stripped) must be one of VALID_SCHEMES = {'http','https','file','raw'}; anything else raises ValueError immediately. Note this check runs before the per-scheme handlers, and unlike the Playwright strategy there is no 'about:' or other special-case scheme.","triggerScenarios":"Passing URLs like 'ftp://...', 'data:text/html,...', 'about:blank', scheme-less strings ('example.com'), or URLs where urlparse yields an unexpected scheme ('raw:' with extra slashes is fine, but 'RAW:' uppercase fails since matching is case-sensitive here).","commonSituations":"Feeding scraped hrefs (mailto:, tel:, javascript:, data:) directly into the HTTP crawler; input lists missing scheme normalization; uppercase scheme variants from user input.","solutions":["Normalize and filter URLs before crawling: lowercase the scheme and require it to be in {'http','https','file','raw'}.","For inline HTML use the raw: prefix ('raw:...'), for local files file://, otherwise http(s).","Drop non-web links (mailto:, tel:, javascript:, ftp:) when building the crawl queue."],"exampleFix":"// before\nawait crawler.crawler_strategy.crawl(\"FTP://example.com/file\")\n\n// after\nfrom urllib.parse import urlparse\nu = url.strip()\nif urlparse(u).scheme.lower() not in (\"http\", \"https\", \"file\", \"raw\"):\n u = \"https://\" + u.lstrip(\"/\")\nawait crawler.crawler_strategy.crawl(u)","handlingStrategy":"type-guard","validationCode":"from urllib.parse import urlparse\n\nVALID = {\"http\", \"https\", \"file\", \"raw\"}\n\ndef crawlable(url: str) -> bool:\n return urlparse(url.strip()).scheme.lower().rstrip(\"/\") in VALID","typeGuard":"from urllib.parse import urlparse\n\nVALID_SCHEMES = frozenset({\"http\", \"https\", \"file\", \"raw\"})\n\ndef is_crawlable_url(url: str) -> bool:\n try:\n return urlparse(url).scheme.lower().rstrip(\"/\") in VALID_SCHEMES\n except ValueError:\n return False","tryCatchPattern":"try:\n await crawler.crawler_strategy.crawl(url)\nexcept ValueError as e:\n if \"Unsupported URL scheme\" in str(e):\n skip(url)","preventionTips":["Filter crawl queues to http/https/file/raw","Lowercase schemes from user input","Reject data:, mailto:, javascript: hrefs at scrape time"],"tags":["url-validation","scheme","http-crawler","input-validation"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}