{"record":{"id":"2034bddbbb8e10eb","repo":"unclecode/crawl4ai","slug":"url-must-start-with-schemes","errorCode":null,"errorMessage":"URL must start with {schemes}","messagePattern":"URL must start with (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"warning","filePath":"deploy/docker/server.py","lineNumber":451,"sourceCode":"    gen = _secrets.token_hex(32)\n    os.environ[\"CRAWL4AI_API_TOKEN\"] = gen\n    logger.warning(\n        \"No CRAWL4AI_API_TOKEN set; generated an ephemeral token for this \"\n        \"loopback session:\\n    CRAWL4AI_API_TOKEN=%s\",\n        gen,\n    )\n\n# ───────────────── URL validation helper ─────────────────\nALLOWED_URL_SCHEMES = (\"http://\", \"https://\")\nALLOWED_URL_SCHEMES_WITH_RAW = (\"http://\", \"https://\", \"raw:\", \"raw://\")\n\n\ndef validate_url_scheme(url: str, allow_raw: bool = False) -> None:\n    \"\"\"Validate URL scheme (LFI) and destination (SSRF).\"\"\"\n    allowed = ALLOWED_URL_SCHEMES_WITH_RAW if allow_raw else ALLOWED_URL_SCHEMES\n    if not url.startswith(allowed):\n        schemes = \", \".join(allowed)\n        raise HTTPException(400, f\"URL must start with {schemes}\")\n    validate_url_destination(url)\n\n\n# ───────────────── safe config‑dump helper ─────────────────\nALLOWED_TYPES = {\n    \"CrawlerRunConfig\": CrawlerRunConfig,\n    \"BrowserConfig\": BrowserConfig,\n}\n\n\ndef _config_from_json(data: dict) -> dict:\n    \"\"\"Validate a {type, params} config under the untrusted trust boundary and\n    echo the normalized result.\n\n    This endpoint is no longer a gadget-construction oracle: only the gated,\n    side-effect-free CrawlerRunConfig/BrowserConfig types may be validated, the\n    untrusted gate raises on forbidden power-fields and disallowed nested types\n    (LLM*, proxy, deep-crawl - which is what would read env/secrets), drops","sourceCodeStart":433,"sourceCodeEnd":469,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/server.py#L433-L469","documentation":"An explicit 400 raised by validate_url_scheme() in the crawl server when a submitted URL does not start with an allowed scheme: 'http://' or 'https://' (plus 'raw:'/'raw://' only when allow_raw is true). This is a deliberate LFI/SSRF guard that rejects file://, data:, and other schemes before the URL is fetched; after the prefix check the URL also goes through destination validation for SSRF.","triggerScenarios":"Submitting a crawl request with url='file:///etc/passwd' (LFI attempt or misconfigured client), url='ftp://example.com', a bare hostname like 'example.com' with no scheme, or a 'raw:' URL to an endpoint that does not pass allow_raw=True.","commonSituations":"Users pasting URLs without the scheme; legacy clients constructing 'raw:...' internal URLs against newer endpoints that no longer allow raw; security scanners probing for file:// traversal; upstream data containing non-HTTP links passed through unnormalized.","solutions":["Normalize URLs client-side to a full absolute http:// or https:// form before submitting (prepend 'https://' for bare hostnames).","For raw-payload URLs, call the endpoint variant that validates with allow_raw=True; otherwise strip the raw: prefix and send the content directly if the API supports it.","Filter or normalize non-HTTP links in upstream datasets (drop mailto:, ftp:, file:) instead of submitting them."],"exampleFix":"# before\nurl = 'example.com/page'          # 400: URL must start with http://, https://\n\n# after\nfrom urllib.parse import urlsplit\np = urlsplit(url)\nif not p.scheme:\n    url = 'https://' + url       # -> https://example.com/page","handlingStrategy":"validation","validationCode":"from urllib.parse import urlsplit\n\nALLOWED = ('http://', 'https://')\n\ndef normalize_url(u: str) -> str:\n    u = u.strip()\n    if not urlsplit(u).scheme:\n        u = 'https://' + u\n    return u\n\ndef url_scheme_ok(u: str, allow_raw: bool = False) -> bool:\n    allowed = ALLOWED + ('raw:', 'raw://') if allow_raw else ALLOWED\n    return u.startswith(allowed)","typeGuard":"from typing import Literal\n\nUrl = str\n\ndef is_http_url(v: str) -> bool:\n    p = urlsplit(v)\n    return p.scheme in ('http', 'https') and bool(p.netloc)","tryCatchPattern":"try:\n    resp = post(f\"{base}/crawl\", json={'url': normalize_url(url)})\nexcept HTTPError as e:\n    if e.response.status_code == 400 and 'URL must start with' in e.response.text:\n        raise ValueError(f'unsupported scheme for {url!r}; only http/https allowed') from e\n    raise","preventionTips":["Always normalize user-supplied URLs to absolute http/https form before submitting.","Filter file://, ftp:, data:, mailto: links out of crawl inputs upstream.","Only send raw: URLs to endpoints documented to accept them (allow_raw=True)."],"tags":["fastapi","validation","security","ssrf","lfi","url"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}