{"record":{"id":"efd5c449e007cb4d","repo":"unclecode/crawl4ai","slug":"url-blocked-efd5c4","errorCode":null,"errorMessage":"URL blocked","messagePattern":"URL blocked","errorType":"validation","errorClass":"ValueError","httpStatus":400,"severity":"error","filePath":"deploy/docker/utils.py","lineNumber":399,"sourceCode":"    return candidates\n\n\ndef validate_webhook_url(url: str) -> None:\n    \"\"\"Reject webhook/crawl URLs targeting non-global networks (SSRF protection).\n\n    Delegates to the single egress rule (egress_broker: reject any resolved IP\n    where not ip.is_global, including v4-mapped/NAT64/6to4/v4-compat embedded\n    forms). The raised message is intentionally opaque - it never echoes the\n    resolved IP or hostname, so this is not a DNS/oracle leak.\n    \"\"\"\n    from egress_broker import resolve_and_pin, EgressBlocked\n    parsed = urlparse(str(url))\n    if not parsed.hostname:\n        raise ValueError(\"URL must have a valid hostname\")\n    try:\n        resolve_and_pin(url)\n    except EgressBlocked:\n        raise ValueError(\"URL blocked\")\n\n\ndef verify_email_domain(email: str) -> bool:\n    try:\n        domain = email.split('@')[1]\n        # Try to resolve MX records for the domain.\n        records = dns.resolver.resolve(domain, 'MX')\n        return True if records else False\n    except Exception as e:\n        return False\n\ndef get_container_memory_percent() -> float:\n    \"\"\"Get actual container memory usage vs limit (cgroup v1/v2 aware).\"\"\"\n    try:\n        # Try cgroup v2 first\n        usage_path = Path(\"/sys/fs/cgroup/memory.current\")\n        limit_path = Path(\"/sys/fs/cgroup/memory.max\")\n        if not usage_path.exists():","sourceCodeStart":381,"sourceCodeEnd":417,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/utils.py#L381-L417","documentation":"Opaque ValueError from validate_webhook_url: egress_broker.resolve_and_pin resolved the URL and the IP was not ip.is_global (private/loopback/reserved, including v4-mapped/NAT64/6to4 forms). The message deliberately omits the IP/hostname so it cannot be used as a DNS oracle.","triggerScenarios":"Webhook or crawl URL pointing at internal infrastructure: http://10.0.0.5/hook, http://169.254.169.254/latest/meta-data (cloud metadata), http://localhost:8000, or a hostname that resolves into RFC1918 space.","commonSituations":"Pointing the crawl-completed webhook at an internal service (n8n, internal API) while running the public/hosted server; SSRF probe attempts (this is the guard working); split-horizon DNS resolving an internal IP for a public-looking name.","solutions":["Host the webhook on a publicly reachable endpoint (public load balancer or a tunnel like ngrok) so the broker's resolution passes","Self-host the crawler with CRAWL4AI_ALLOW_INTERNAL_URLS=true when both crawler and webhook live inside the same private network","If you believe the target is public, verify from the server container: dig +short <host> must return a global IP","Do not attempt to extract the blocked IP from the error - it is intentionally not included"],"exampleFix":"# before\n{\"webhook\": {\"url\": \"http://10.1.2.3/crawl-done\"}}\n\n# after - expose it publicly or relax on self-host\n# public: https://hooks.mycompany.com/crawl-done\n# self-hosted: docker run -e CRAWL4AI_ALLOW_INTERNAL_URLS=true ...","handlingStrategy":"validation","validationCode":"import ipaddress, socket\nfrom urllib.parse import urlparse\n\ndef webhook_url_is_public(url: str) -> bool:\n    host = urlparse(url).hostname or \"\"\n    try:\n        return all(ipaddress.ip_address(i[4][0]).is_global\n                   for i in socket.getaddrinfo(host, None))\n    except socket.gaierror:\n        return False","typeGuard":"def is_safe_webhook_config(cfg: dict) -> bool:\n    url = cfg.get(\"url\", \"\")\n    return is_absolute_http_url(url) and webhook_url_is_public(url)","tryCatchPattern":"try:\n    validate_webhook_url(url)\nexcept ValueError as e:\n    if str(e) == \"URL blocked\":\n        # opaque by design: switch to public endpoint or self-host\n        url = public_tunnel(url)","preventionTips":["Point webhooks at a public ingress (LB or tunnel) from day one","When self-hosting in a private network, set CRAWL4AI_ALLOW_INTERNAL_URLS=true deliberately and keep the instance trusted-network-only","Never treat the opaque message as containing diagnostic info - resolve the target yourself"],"tags":["crawl4ai","ssrf","webhook","security","network"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}