{"record":{"id":"289162a1aaf4b044","repo":"unclecode/crawl4ai","slug":"url-must-start-with-http-https-file","errorCode":null,"errorMessage":"URL must start with 'http://', 'https://', 'file://', or 'raw:'","messagePattern":"URL must start with 'http://', 'https://', 'file://', or 'raw:'","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_crawler_strategy.py","lineNumber":510,"sourceCode":" with open(local_file_path, \"r\", encoding=\"utf-8\") as f:\n html = f.read()\n else:\n # Process raw HTML content (raw:// or raw:)\n html = url[6:] if url.startswith(\"raw://\") else url[4:]\n\n return AsyncCrawlResponse(\n html=html,\n response_headers=response_headers,\n status_code=status_code,\n screenshot=None,\n pdf_data=None,\n mhtml_data=None,\n get_delayed_content=None,\n # For raw:/file:// URLs, use base_url if provided; don't fall back to the raw content\n redirected_url=config.base_url,\n )\n else:\n raise ValueError(\n \"URL must start with 'http://', 'https://', 'file://', or 'raw:'\"\n )\n\n async def _crawl_web(\n self, url: str, config: CrawlerRunConfig\n ) -> AsyncCrawlResponse:\n \"\"\"\n Internal method to crawl web URLs with the specified configuration.\n Includes optional network and console capturing.\n\n Args:\n url (str): The web URL to crawl\n config (CrawlerRunConfig): Configuration object controlling the crawl behavior\n\n Returns:\n AsyncCrawlResponse: The response containing HTML, headers, status code, and optional data\n \"\"\"\n config.url = url","sourceCodeStart":492,"sourceCodeEnd":528,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_crawler_strategy.py#L492-L528","documentation":"Raised by the browser crawler's crawl entry point when the URL does not begin with http://, https://, file://, or raw:. The crawler supports only these schemes; anything else (ftp://, about:, chrome://, plain strings with no scheme, or malformed URLs) is rejected before any crawling starts.","triggerScenarios":"Passing a URL string with no scheme ('example.com/page'), a non-supported scheme ('ftp://...', 'data:text/html,...'), or a scheme with different casing/casing or trailing whitespace. The check happens in the else-branch of the scheme dispatch in PlaywrightCrawlerStrategy.crawl().","commonSituations":"User input not normalized (missing https:// prefix); feeding data: URIs or mailto: links scraped from pages into arun(); passing raw HTML without the raw:// prefix; URLs with leading/trailing whitespace from CSV/spreadsheet imports.","solutions":["Normalize the URL before crawling: strip whitespace and prepend 'https://' if no scheme is present.","Wrap raw HTML strings in the raw:// prefix (e.g. 'raw:...') instead of passing them bare.","Filter or reject non-http(s)/file/raw links when building a crawl queue from scraped hrefs.","URL-encode or verify the input with urllib.parse.urlparse and check .scheme before calling the crawler."],"exampleFix":"// before\nawait crawler.arun(url=\"example.com/docs\")\n\n// after\nurl = \"example.com/docs\".strip()\nif not url.startswith((\"http://\", \"https://\", \"file://\", \"raw:\", \"raw://\")):\n url = \"https://\" + url\nawait crawler.arun(url=url)","handlingStrategy":"validation","validationCode":"ALLOWED = (\"http://\", \"https://\", \"file://\", \"raw:\", \"raw://\")\n\ndef normalize_url(url: str) -> str:\n u = url.strip()\n if not u.startswith(ALLOWED):\n u = \"https://\" + u\n return u","typeGuard":null,"tryCatchPattern":"try:\n await crawler.arun(url)\nexcept ValueError as e:\n if \"URL must start with\" in str(e):\n logger.warning(f\"invalid scheme, skipping: {url}\")","preventionTips":["Normalize URLs at queue-build time","Wrap inline HTML in raw: prefix","Filter scraped hrefs to http(s) only"],"tags":["url-validation","scheme","input-validation"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}