unclecode/crawl4ai · error · ValueError

URL must start with 'http://', 'https://', 'file://', or 'ra

Error message

URL must start with 'http://', 'https://', 'file://', or 'raw:'

What it means

Raised by the browser crawler's crawl entry point when the URL does not begin with http://, https://, file://, or raw:. The crawler supports only these schemes; anything else (ftp://, about:, chrome://, plain strings with no scheme, or malformed URLs) is rejected before any crawling starts.

Source

Thrown at crawl4ai/async_crawler_strategy.py:510

                with open(local_file_path, "r", encoding="utf-8") as f:
                    html = f.read()
            else:
                # Process raw HTML content (raw:// or raw:)
                html = url[6:] if url.startswith("raw://") else url[4:]

            return AsyncCrawlResponse(
                html=html,
                response_headers=response_headers,
                status_code=status_code,
                screenshot=None,
                pdf_data=None,
                mhtml_data=None,
                get_delayed_content=None,
                # For raw:/file:// URLs, use base_url if provided; don't fall back to the raw content
                redirected_url=config.base_url,
            )
        else:
            raise ValueError(
                "URL must start with 'http://', 'https://', 'file://', or 'raw:'"
            )

    async def _crawl_web(
        self, url: str, config: CrawlerRunConfig
    ) -> AsyncCrawlResponse:
        """
        Internal method to crawl web URLs with the specified configuration.
        Includes optional network and console capturing.

        Args:
            url (str): The web URL to crawl
            config (CrawlerRunConfig): Configuration object controlling the crawl behavior

        Returns:
            AsyncCrawlResponse: The response containing HTML, headers, status code, and optional data
        """
        config.url = url

View on GitHub (pinned to 7e80152142)

Solutions

  1. Normalize the URL before crawling: strip whitespace and prepend 'https://' if no scheme is present.
  2. Wrap raw HTML strings in the raw:// prefix (e.g. 'raw:<html>...') instead of passing them bare.
  3. Filter or reject non-http(s)/file/raw links when building a crawl queue from scraped hrefs.
  4. URL-encode or verify the input with urllib.parse.urlparse and check .scheme before calling the crawler.

Example fix

// before
await crawler.arun(url="example.com/docs")

// after
url = "example.com/docs".strip()
if not url.startswith(("http://", "https://", "file://", "raw:", "raw://")):
    url = "https://" + url
await crawler.arun(url=url)
Defensive patterns

Strategy: validation

Validate before calling

ALLOWED = ("http://", "https://", "file://", "raw:", "raw://")

def normalize_url(url: str) -> str:
    u = url.strip()
    if not u.startswith(ALLOWED):
        u = "https://" + u
    return u

Try / catch

try:
    await crawler.arun(url)
except ValueError as e:
    if "URL must start with" in str(e):
        logger.warning(f"invalid scheme, skipping: {url}")

Prevention

When it happens

Trigger: Passing a URL string with no scheme ('example.com/page'), a non-supported scheme ('ftp://...', 'data:text/html,...'), or a scheme with different casing/casing or trailing whitespace. The check happens in the else-branch of the scheme dispatch in PlaywrightCrawlerStrategy.crawl().

Common situations: User input not normalized (missing https:// prefix); feeding data: URIs or mailto: links scraped from pages into arun(); passing raw HTML without the raw:// prefix; URLs with leading/trailing whitespace from CSV/spreadsheet imports.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/289162a1aaf4b044. Report an issue: GitHub.