unclecode/crawl4ai · error · ValueError

URL must have a valid hostname

Error message

URL must have a valid hostname

What it means

ValueError from validate_webhook_url when urlparse(str(url)).hostname is empty - the URL has no parsable host (e.g. 'http://', '/hook', 'not-a-url'). It is raised before any DNS resolution; via validate_url_destination it also surfaces to callers validating crawl URLs.

Source

Thrown at deploy/docker/utils.py:395

        else:
            as_int = int(ip)
            if 0 < as_int < 2**32:
                candidates.append(ipaddress.IPv4Address(as_int))
    return candidates


def validate_webhook_url(url: str) -> None:
    """Reject webhook/crawl URLs targeting non-global networks (SSRF protection).

    Delegates to the single egress rule (egress_broker: reject any resolved IP
    where not ip.is_global, including v4-mapped/NAT64/6to4/v4-compat embedded
    forms). The raised message is intentionally opaque - it never echoes the
    resolved IP or hostname, so this is not a DNS/oracle leak.
    """
    from egress_broker import resolve_and_pin, EgressBlocked
    parsed = urlparse(str(url))
    if not parsed.hostname:
        raise ValueError("URL must have a valid hostname")
    try:
        resolve_and_pin(url)
    except EgressBlocked:
        raise ValueError("URL blocked")


def verify_email_domain(email: str) -> bool:
    try:
        domain = email.split('@')[1]
        # Try to resolve MX records for the domain.
        records = dns.resolver.resolve(domain, 'MX')
        return True if records else False
    except Exception as e:
        return False

def get_container_memory_percent() -> float:
    """Get actual container memory usage vs limit (cgroup v1/v2 aware)."""
    try:

View on GitHub (pinned to 7e80152142)

Solutions

1. Log/inspect the exact URL string before sending it
2. Fix the source of the URL: require scheme + host, e.g. https://hooks.example.com/endpoint
3. If it comes from env config, fail fast at startup when the var is empty rather than at request time

Example fix

# before
url = os.environ.get("WEBHOOK_URL", "")

# after
from urllib.parse import urlparse
url = os.environ["WEBHOOK_URL"]
assert urlparse(url).hostname, "WEBHOOK_URL must be an absolute http(s) URL"

from urllib.parse import urlparse

def has_hostname(url: str) -> bool:
    try:
        return bool(urlparse(str(url).strip()).hostname)
    except ValueError:
        return False

def is_absolute_http_url(url) -> bool:
    p = urlparse(str(url))
    return p.scheme in ("http", "https") and bool(p.hostname)

try:
    validate_webhook_url(url)
except ValueError as e:
    if "hostname" in str(e):
        url = f"https://{url}"  # repair scheme-less input, then revalidate
        validate_webhook_url(url)

When it happens

Trigger: Registering a webhook or submitting a crawl URL like 'http://', 'webhook', or a bare path with no scheme/host, or a value with whitespace/control characters that breaks parsing.

Common situations: Webhook URL loaded from an env var that is unset (empty string) or misconfigured; URL built by string concatenation where the host segment ended up empty; trailing punctuation or copy-paste artifacts from docs.

Related errors

• too many webhook headers
• invalid webhook header name: {name!r}
• webhook header not allowed: {name}
• invalid value for webhook header {name}
• Hook '{hook_name}' must be a callable function, got {type(ho

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/30ed94a6d123a65e. Report an issue: GitHub.