unclecode/crawl4ai · error · ValueError

URL blocked

Error message

URL blocked

What it means

Opaque ValueError from validate_webhook_url: egress_broker.resolve_and_pin resolved the URL and the IP was not ip.is_global (private/loopback/reserved, including v4-mapped/NAT64/6to4 forms). The message deliberately omits the IP/hostname so it cannot be used as a DNS oracle.

Source

Thrown at deploy/docker/utils.py:399

    return candidates


def validate_webhook_url(url: str) -> None:
    """Reject webhook/crawl URLs targeting non-global networks (SSRF protection).

    Delegates to the single egress rule (egress_broker: reject any resolved IP
    where not ip.is_global, including v4-mapped/NAT64/6to4/v4-compat embedded
    forms). The raised message is intentionally opaque - it never echoes the
    resolved IP or hostname, so this is not a DNS/oracle leak.
    """
    from egress_broker import resolve_and_pin, EgressBlocked
    parsed = urlparse(str(url))
    if not parsed.hostname:
        raise ValueError("URL must have a valid hostname")
    try:
        resolve_and_pin(url)
    except EgressBlocked:
        raise ValueError("URL blocked")


def verify_email_domain(email: str) -> bool:
    try:
        domain = email.split('@')[1]
        # Try to resolve MX records for the domain.
        records = dns.resolver.resolve(domain, 'MX')
        return True if records else False
    except Exception as e:
        return False

def get_container_memory_percent() -> float:
    """Get actual container memory usage vs limit (cgroup v1/v2 aware)."""
    try:
        # Try cgroup v2 first
        usage_path = Path("/sys/fs/cgroup/memory.current")
        limit_path = Path("/sys/fs/cgroup/memory.max")
        if not usage_path.exists():

View on GitHub (pinned to 7e80152142)

Solutions

1. Host the webhook on a publicly reachable endpoint (public load balancer or a tunnel like ngrok) so the broker's resolution passes
2. Self-host the crawler with CRAWL4AI_ALLOW_INTERNAL_URLS=true when both crawler and webhook live inside the same private network
3. If you believe the target is public, verify from the server container: dig +short <host> must return a global IP
4. Do not attempt to extract the blocked IP from the error - it is intentionally not included

Example fix

# before
{"webhook": {"url": "http://10.1.2.3/crawl-done"}}

# after - expose it publicly or relax on self-host
# public: https://hooks.mycompany.com/crawl-done
# self-hosted: docker run -e CRAWL4AI_ALLOW_INTERNAL_URLS=true ...

import ipaddress, socket
from urllib.parse import urlparse

def webhook_url_is_public(url: str) -> bool:
    host = urlparse(url).hostname or ""
    try:
        return all(ipaddress.ip_address(i[4][0]).is_global
                   for i in socket.getaddrinfo(host, None))
    except socket.gaierror:
        return False

def is_safe_webhook_config(cfg: dict) -> bool:
    url = cfg.get("url", "")
    return is_absolute_http_url(url) and webhook_url_is_public(url)

try:
    validate_webhook_url(url)
except ValueError as e:
    if str(e) == "URL blocked":
        # opaque by design: switch to public endpoint or self-host
        url = public_tunnel(url)

When it happens

Trigger: Webhook or crawl URL pointing at internal infrastructure: http://10.0.0.5/hook, http://169.254.169.254/latest/meta-data (cloud metadata), http://localhost:8000, or a hostname that resolves into RFC1918 space.

Common situations: Pointing the crawl-completed webhook at an internal service (n8n, internal API) while running the public/hosted server; SSRF probe attempts (this is the guard working); split-horizon DNS resolving an internal IP for a public-looking name.

Related errors

• URL blocked (SSRF protection): {e}
• webhook header not allowed: {name}
• URL must start with {schemes}
• URL must have a valid hostname
• too many webhook headers

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/efd5c449e007cb4d. Report an issue: GitHub.