{"record":{"id":"b284c146981ba8c7","repo":"unclecode/crawl4ai","slug":"url-blocked-ssrf-protection-e","errorCode":null,"errorMessage":"URL blocked (SSRF protection): {e}","messagePattern":"URL blocked \\(SSRF protection\\): (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"deploy/docker/utils.py","lineNumber":366,"sourceCode":"}\n\n\nALLOW_INTERNAL_URLS = os.environ.get(\"CRAWL4AI_ALLOW_INTERNAL_URLS\", \"false\").lower() == \"true\"\n\n\ndef validate_url_destination(url: str) -> None:\n \"\"\"Block crawl URLs targeting internal/private networks (SSRF protection).\n Skipped when CRAWL4AI_ALLOW_INTERNAL_URLS=true.\n Skipped for raw: URLs (inline HTML, no network fetch).\"\"\"\n if ALLOW_INTERNAL_URLS:\n return\n if str(url).startswith((\"raw:\", \"raw://\")):\n return\n try:\n validate_webhook_url(url)\n except ValueError as e:\n from fastapi import HTTPException\n raise HTTPException(status_code=400, detail=f\"URL blocked (SSRF protection): {e}\")\n\n\ndef _expand_ip_candidates(ip):\n \"\"\"Return [ip] plus any IPv4 form wrapped inside the IPv6 address.\n SSRF guards must check the unwrapped form because ::ffff:127.0.0.1 and\n ::127.0.0.1 route to 127.0.0.1 but would not match IPv4 blocklists directly.\"\"\"\n candidates = [ip]\n if isinstance(ip, ipaddress.IPv6Address):\n if ip.ipv4_mapped is not None:\n candidates.append(ip.ipv4_mapped)\n else:\n as_int = int(ip)\n if 0 < as_int < 2**32:\n candidates.append(ipaddress.IPv4Address(as_int))\n return candidates\n\n\ndef validate_webhook_url(url: str) -> None:","sourceCodeStart":348,"sourceCodeEnd":384,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/utils.py#L348-L384","documentation":"HTTP 400 from validate_url_destination: the crawl URL resolved to a non-global IP (private, loopback, link-local, or a v4-mapped/NAT64 embedded form), so the SSRF egress guard rejected it. Bypassed only when CRAWL4AI_ALLOW_INTERNAL_URLS=true or the URL starts with raw:.","triggerScenarios":"POST /crawl with a URL whose hostname resolves to 10.x/192.168.x/127.0.0.1/169.254.x/::1 (or ::ffff:10.x etc.); also http://localhost:8080, http://kubernetes.default.svc inside a cluster, or a public name that DNS-rebinds to an internal IP.","commonSituations":"Using the hosted server to fetch a local dev site; crawling internal service names in Kubernetes; testing against localhost - none of which the public server permits by design. Sometimes a corporate DNS returns an internal IP for an apparently public hostname.","solutions":["Crawl the internal target from a self-hosted server started with CRAWL4AI_ALLOW_INTERNAL_URLS=true (only on trusted networks)","For inline HTML you control, use the raw: URL scheme which skips the network fetch and the guard","For local testing, run the library in-process (AsyncWebCrawler) instead of through the deployed server","If the URL is genuinely public but blocked, check your DNS - the name may be resolving internally (split-horizon); fix resolver config or use a truly public hostname"],"exampleFix":"# before\nPOST /crawl {\"urls\": [\"http://localhost:3000/page\"]}\n\n# after - self-hosted with the guard relaxed\ndocker run -e CRAWL4AI_ALLOW_INTERNAL_URLS=true -p 11235:11235 unclecode/crawl4ai\n# or inline HTML on the hosted server\nPOST /crawl {\"urls\": [\"raw:hi\"]}","handlingStrategy":"validation","validationCode":"import ipaddress, socket\nfrom urllib.parse import urlparse\n\ndef is_public_target(url: str) -> bool:\n host = urlparse(url).hostname or \"\"\n try:\n infos = socket.getaddrinfo(host, None)\n except socket.gaierror:\n return False\n return all(ipaddress.ip_address(i[4][0]).is_global for i in infos)","typeGuard":"def is_crawlable_url(url: str) -> bool:\n return bool(urlparse(url).hostname) and is_public_target(url)","tryCatchPattern":"try:\n resp = requests.post(f\"{S}/crawl\", json={\"urls\": [u]})\n resp.raise_for_status()\nexcept requests.HTTPError as e:\n if e.response.status_code == 400 and \"SSRF\" in e.response.text:\n skip_or_tunnel(u) # route via self-hosted instance or public mirror","preventionTips":["Run the same is_global check the server uses before submitting batches of user-supplied URLs","For local/intranet targets, always self-host with CRAWL4AI_ALLOW_INTERNAL_URLS=true rather than fighting the guard","Use raw: URLs for pure HTML-processing calls so no network egress is attempted"],"tags":["crawl4ai","ssrf","http-400","security","network"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}