unclecode/crawl4ai · warning · HTTPException

URL must start with {schemes}

Error message

URL must start with {schemes}

What it means

An explicit 400 raised by validate_url_scheme() in the crawl server when a submitted URL does not start with an allowed scheme: 'http://' or 'https://' (plus 'raw:'/'raw://' only when allow_raw is true). This is a deliberate LFI/SSRF guard that rejects file://, data:, and other schemes before the URL is fetched; after the prefix check the URL also goes through destination validation for SSRF.

Source

Thrown at deploy/docker/server.py:451

    gen = _secrets.token_hex(32)
    os.environ["CRAWL4AI_API_TOKEN"] = gen
    logger.warning(
        "No CRAWL4AI_API_TOKEN set; generated an ephemeral token for this "
        "loopback session:\n    CRAWL4AI_API_TOKEN=%s",
        gen,
    )

# ───────────────── URL validation helper ─────────────────
ALLOWED_URL_SCHEMES = ("http://", "https://")
ALLOWED_URL_SCHEMES_WITH_RAW = ("http://", "https://", "raw:", "raw://")


def validate_url_scheme(url: str, allow_raw: bool = False) -> None:
    """Validate URL scheme (LFI) and destination (SSRF)."""
    allowed = ALLOWED_URL_SCHEMES_WITH_RAW if allow_raw else ALLOWED_URL_SCHEMES
    if not url.startswith(allowed):
        schemes = ", ".join(allowed)
        raise HTTPException(400, f"URL must start with {schemes}")
    validate_url_destination(url)


# ───────────────── safe config‑dump helper ─────────────────
ALLOWED_TYPES = {
    "CrawlerRunConfig": CrawlerRunConfig,
    "BrowserConfig": BrowserConfig,
}


def _config_from_json(data: dict) -> dict:
    """Validate a {type, params} config under the untrusted trust boundary and
    echo the normalized result.

    This endpoint is no longer a gadget-construction oracle: only the gated,
    side-effect-free CrawlerRunConfig/BrowserConfig types may be validated, the
    untrusted gate raises on forbidden power-fields and disallowed nested types
    (LLM*, proxy, deep-crawl - which is what would read env/secrets), drops

View on GitHub (pinned to 7e80152142)

Solutions

1. Normalize URLs client-side to a full absolute http:// or https:// form before submitting (prepend 'https://' for bare hostnames).
2. For raw-payload URLs, call the endpoint variant that validates with allow_raw=True; otherwise strip the raw: prefix and send the content directly if the API supports it.
3. Filter or normalize non-HTTP links in upstream datasets (drop mailto:, ftp:, file:) instead of submitting them.

Example fix

# before
url = 'example.com/page'          # 400: URL must start with http://, https://

# after
from urllib.parse import urlsplit
p = urlsplit(url)
if not p.scheme:
    url = 'https://' + url       # -> https://example.com/page

from urllib.parse import urlsplit

ALLOWED = ('http://', 'https://')

def normalize_url(u: str) -> str:
    u = u.strip()
    if not urlsplit(u).scheme:
        u = 'https://' + u
    return u

def url_scheme_ok(u: str, allow_raw: bool = False) -> bool:
    allowed = ALLOWED + ('raw:', 'raw://') if allow_raw else ALLOWED
    return u.startswith(allowed)

from typing import Literal

Url = str

def is_http_url(v: str) -> bool:
    p = urlsplit(v)
    return p.scheme in ('http', 'https') and bool(p.netloc)

try:
    resp = post(f"{base}/crawl", json={'url': normalize_url(url)})
except HTTPError as e:
    if e.response.status_code == 400 and 'URL must start with' in e.response.text:
        raise ValueError(f'unsupported scheme for {url!r}; only http/https allowed') from e
    raise

When it happens

Trigger: Submitting a crawl request with url='file:///etc/passwd' (LFI attempt or misconfigured client), url='ftp://example.com', a bare hostname like 'example.com' with no scheme, or a 'raw:' URL to an endpoint that does not pass allow_raw=True.

Common situations: Users pasting URLs without the scheme; legacy clients constructing 'raw:...' internal URLs against newer endpoints that no longer allow raw; security scanners probing for file:// traversal; upstream data containing non-HTTP links passed through unnormalized.

Related errors

• type must be 'CrawlerRunConfig' or 'BrowserConfig'
• LLMConfig.api_token may not reference an environment variabl
• invalid header name {name!r}
• control characters in value for header {name!r}
• Invalid metric: {metric}. Must be one of: memory, requests,

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/2034bddbbb8e10eb. Report an issue: GitHub.