unclecode/crawl4ai · error · HTTPException

URL blocked (SSRF protection): {e}

Error message

URL blocked (SSRF protection): {e}

What it means

HTTP 400 from validate_url_destination: the crawl URL resolved to a non-global IP (private, loopback, link-local, or a v4-mapped/NAT64 embedded form), so the SSRF egress guard rejected it. Bypassed only when CRAWL4AI_ALLOW_INTERNAL_URLS=true or the URL starts with raw:.

Source

Thrown at deploy/docker/utils.py:366

}


ALLOW_INTERNAL_URLS = os.environ.get("CRAWL4AI_ALLOW_INTERNAL_URLS", "false").lower() == "true"


def validate_url_destination(url: str) -> None:
    """Block crawl URLs targeting internal/private networks (SSRF protection).
    Skipped when CRAWL4AI_ALLOW_INTERNAL_URLS=true.
    Skipped for raw: URLs (inline HTML, no network fetch)."""
    if ALLOW_INTERNAL_URLS:
        return
    if str(url).startswith(("raw:", "raw://")):
        return
    try:
        validate_webhook_url(url)
    except ValueError as e:
        from fastapi import HTTPException
        raise HTTPException(status_code=400, detail=f"URL blocked (SSRF protection): {e}")


def _expand_ip_candidates(ip):
    """Return [ip] plus any IPv4 form wrapped inside the IPv6 address.
    SSRF guards must check the unwrapped form because ::ffff:127.0.0.1 and
    ::127.0.0.1 route to 127.0.0.1 but would not match IPv4 blocklists directly."""
    candidates = [ip]
    if isinstance(ip, ipaddress.IPv6Address):
        if ip.ipv4_mapped is not None:
            candidates.append(ip.ipv4_mapped)
        else:
            as_int = int(ip)
            if 0 < as_int < 2**32:
                candidates.append(ipaddress.IPv4Address(as_int))
    return candidates


def validate_webhook_url(url: str) -> None:

View on GitHub (pinned to 7e80152142)

Solutions

  1. Crawl the internal target from a self-hosted server started with CRAWL4AI_ALLOW_INTERNAL_URLS=true (only on trusted networks)
  2. For inline HTML you control, use the raw:<html> URL scheme which skips the network fetch and the guard
  3. For local testing, run the library in-process (AsyncWebCrawler) instead of through the deployed server
  4. If the URL is genuinely public but blocked, check your DNS - the name may be resolving internally (split-horizon); fix resolver config or use a truly public hostname

Example fix

# before
POST /crawl {"urls": ["http://localhost:3000/page"]}

# after - self-hosted with the guard relaxed
docker run -e CRAWL4AI_ALLOW_INTERNAL_URLS=true -p 11235:11235 unclecode/crawl4ai
# or inline HTML on the hosted server
POST /crawl {"urls": ["raw:<html><body>hi</body></html>"]}
Defensive patterns

Strategy: validation

Validate before calling

import ipaddress, socket
from urllib.parse import urlparse

def is_public_target(url: str) -> bool:
    host = urlparse(url).hostname or ""
    try:
        infos = socket.getaddrinfo(host, None)
    except socket.gaierror:
        return False
    return all(ipaddress.ip_address(i[4][0]).is_global for i in infos)

Type guard

def is_crawlable_url(url: str) -> bool:
    return bool(urlparse(url).hostname) and is_public_target(url)

Try / catch

try:
    resp = requests.post(f"{S}/crawl", json={"urls": [u]})
    resp.raise_for_status()
except requests.HTTPError as e:
    if e.response.status_code == 400 and "SSRF" in e.response.text:
        skip_or_tunnel(u)  # route via self-hosted instance or public mirror

Prevention

When it happens

Trigger: POST /crawl with a URL whose hostname resolves to 10.x/192.168.x/127.0.0.1/169.254.x/::1 (or ::ffff:10.x etc.); also http://localhost:8080, http://kubernetes.default.svc inside a cluster, or a public name that DNS-rebinds to an internal IP.

Common situations: Using the hosted server to fetch a local dev site; crawling internal service names in Kubernetes; testing against localhost - none of which the public server permits by design. Sometimes a corporate DNS returns an internal IP for an apparently public hostname.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/b284c146981ba8c7. Report an issue: GitHub.