unclecode/crawl4ai · error · ValueError

Unsupported URL scheme: {scheme}

Error message

Unsupported URL scheme: {scheme}

What it means

Entry-point validation of the HTTP-mode crawler: the URL's scheme (from urlparse, trailing '/' stripped) must be one of VALID_SCHEMES = {'http','https','file','raw'}; anything else raises ValueError immediately. Note this check runs before the per-scheme handlers, and unlike the Playwright strategy there is no 'about:' or other special-case scheme.

Source

Thrown at crawl4ai/async_crawler_strategy.py:2810

                raise ConnectionTimeoutError(f"Request timed out: {str(e)}")
            
            except Exception as e:
                await self.hooks['on_error'](e)
                raise HTTPCrawlerError(f"HTTP request failed: {str(e)}")

    async def crawl(
        self, 
        url: str, 
        config: Optional[CrawlerRunConfig] = None, 
        **kwargs
    ) -> AsyncCrawlResponse:
        config = config or CrawlerRunConfig.from_kwargs(kwargs)
        
        parsed = urlparse(url)
        scheme = parsed.scheme.rstrip('/')
        
        if scheme not in self.VALID_SCHEMES:
            raise ValueError(f"Unsupported URL scheme: {scheme}")
            
        try:
            if scheme == 'file':
                return await self._handle_file(parsed.path)
            elif scheme == 'raw':
                # Don't use parsed.path - urlparse truncates at '#' which is common in CSS
                # Strip prefix directly: "raw://" (6 chars) or "raw:" (4 chars)
                raw_content = url[6:] if url.startswith("raw://") else url[4:]
                return await self._handle_raw(raw_content, base_url=config.base_url)
            else:  # http or https
                return await self._handle_http(url, config)
                
        except Exception as e:
            if self.logger:
                self.logger.error(
                    message="Crawl failed: {error}",
                    tag="CRAWL",
                    params={"error": str(e), "url": url}

View on GitHub (pinned to 7e80152142)

Solutions

  1. Normalize and filter URLs before crawling: lowercase the scheme and require it to be in {'http','https','file','raw'}.
  2. For inline HTML use the raw: prefix ('raw:<html>...'), for local files file://, otherwise http(s).
  3. Drop non-web links (mailto:, tel:, javascript:, ftp:) when building the crawl queue.

Example fix

// before
await crawler.crawler_strategy.crawl("FTP://example.com/file")

// after
from urllib.parse import urlparse
u = url.strip()
if urlparse(u).scheme.lower() not in ("http", "https", "file", "raw"):
    u = "https://" + u.lstrip("/")
await crawler.crawler_strategy.crawl(u)
Defensive patterns

Strategy: type-guard

Validate before calling

from urllib.parse import urlparse

VALID = {"http", "https", "file", "raw"}

def crawlable(url: str) -> bool:
    return urlparse(url.strip()).scheme.lower().rstrip("/") in VALID

Type guard

from urllib.parse import urlparse

VALID_SCHEMES = frozenset({"http", "https", "file", "raw"})

def is_crawlable_url(url: str) -> bool:
    try:
        return urlparse(url).scheme.lower().rstrip("/") in VALID_SCHEMES
    except ValueError:
        return False

Try / catch

try:
    await crawler.crawler_strategy.crawl(url)
except ValueError as e:
    if "Unsupported URL scheme" in str(e):
        skip(url)

Prevention

When it happens

Trigger: Passing URLs like 'ftp://...', 'data:text/html,...', 'about:blank', scheme-less strings ('example.com'), or URLs where urlparse yields an unexpected scheme ('raw:' with extra slashes is fine, but 'RAW:' uppercase fails since matching is case-sensitive here).

Common situations: Feeding scraped hrefs (mailto:, tel:, javascript:, data:) directly into the HTTP crawler; input lists missing scheme normalization; uppercase scheme variants from user input.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/3a5a502a3cb516a6. Report an issue: GitHub.