unclecode/crawl4ai · error · EgressBlocked

URL blocked

Error message

URL blocked

What it means

Error "URL blocked" thrown in unclecode/crawl4ai.

Source

Thrown at deploy/docker/egress_broker.py:96

        elif ip in _6TO4:
            forms.append(ipaddress.IPv4Address((int(ip) >> 80) & 0xFFFFFFFF))
    return forms


def is_forbidden_ip(ip_str: str) -> bool:
    """True if the IP (or any embedded transition form) is not globally routable."""
    try:
        ip = ipaddress.ip_address(ip_str)
    except ValueError:
        return True
    return any(not form.is_global for form in _embedded_v4_forms(ip))


def _resolve(host: str, port: int):
    try:
        return socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)
    except socket.gaierror:
        raise EgressBlocked()


def assert_host_allowed(host: str, port: int = 0) -> None:
    """Resolve `host` and reject if ANY answer is non-global. Opaque on failure."""
    if ALLOW_INTERNAL:
        return
    if not host:
        raise EgressBlocked()
    low = host.lower()
    if low in _BLOCKED_HOSTNAMES or low.startswith("host.docker.internal"):
        raise EgressBlocked()
    for *_, sockaddr in _resolve(host, port):
        if is_forbidden_ip(sockaddr[0]):
            raise EgressBlocked()


def resolve_and_pin(url: str) -> PinnedTarget:
    """Resolve `url` once, reject if any answer is non-global, and pin one IP.

View on GitHub (pinned to 7e80152142)

Solutions

1. Use a URL that is permitted by the egress policy (public http/https hosts only).
2. If the host must be allowed, update the egress allowlist configuration on the server.

Example fix

# Avoid private/loopback IPs; crawl public URLs only

When it happens

Trigger: Thrown at deploy/docker/egress_broker.py:96 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/8d8d05ed97d9f561. Report an issue: GitHub.