{"record":{"id":"30ed94a6d123a65e","repo":"unclecode/crawl4ai","slug":"url-must-have-a-valid-hostname","errorCode":null,"errorMessage":"URL must have a valid hostname","messagePattern":"URL must have a valid hostname","errorType":"validation","errorClass":"ValueError","httpStatus":400,"severity":"error","filePath":"deploy/docker/utils.py","lineNumber":395,"sourceCode":"        else:\n            as_int = int(ip)\n            if 0 < as_int < 2**32:\n                candidates.append(ipaddress.IPv4Address(as_int))\n    return candidates\n\n\ndef validate_webhook_url(url: str) -> None:\n    \"\"\"Reject webhook/crawl URLs targeting non-global networks (SSRF protection).\n\n    Delegates to the single egress rule (egress_broker: reject any resolved IP\n    where not ip.is_global, including v4-mapped/NAT64/6to4/v4-compat embedded\n    forms). The raised message is intentionally opaque - it never echoes the\n    resolved IP or hostname, so this is not a DNS/oracle leak.\n    \"\"\"\n    from egress_broker import resolve_and_pin, EgressBlocked\n    parsed = urlparse(str(url))\n    if not parsed.hostname:\n        raise ValueError(\"URL must have a valid hostname\")\n    try:\n        resolve_and_pin(url)\n    except EgressBlocked:\n        raise ValueError(\"URL blocked\")\n\n\ndef verify_email_domain(email: str) -> bool:\n    try:\n        domain = email.split('@')[1]\n        # Try to resolve MX records for the domain.\n        records = dns.resolver.resolve(domain, 'MX')\n        return True if records else False\n    except Exception as e:\n        return False\n\ndef get_container_memory_percent() -> float:\n    \"\"\"Get actual container memory usage vs limit (cgroup v1/v2 aware).\"\"\"\n    try:","sourceCodeStart":377,"sourceCodeEnd":413,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/utils.py#L377-L413","documentation":"ValueError from validate_webhook_url when urlparse(str(url)).hostname is empty - the URL has no parsable host (e.g. 'http://', '/hook', 'not-a-url'). It is raised before any DNS resolution; via validate_url_destination it also surfaces to callers validating crawl URLs.","triggerScenarios":"Registering a webhook or submitting a crawl URL like 'http://', 'webhook', or a bare path with no scheme/host, or a value with whitespace/control characters that breaks parsing.","commonSituations":"Webhook URL loaded from an env var that is unset (empty string) or misconfigured; URL built by string concatenation where the host segment ended up empty; trailing punctuation or copy-paste artifacts from docs.","solutions":["Log/inspect the exact URL string before sending it","Fix the source of the URL: require scheme + host, e.g. https://hooks.example.com/endpoint","If it comes from env config, fail fast at startup when the var is empty rather than at request time"],"exampleFix":"# before\nurl = os.environ.get(\"WEBHOOK_URL\", \"\")\n\n# after\nfrom urllib.parse import urlparse\nurl = os.environ[\"WEBHOOK_URL\"]\nassert urlparse(url).hostname, \"WEBHOOK_URL must be an absolute http(s) URL\"","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\ndef has_hostname(url: str) -> bool:\n    try:\n        return bool(urlparse(str(url).strip()).hostname)\n    except ValueError:\n        return False","typeGuard":"def is_absolute_http_url(url) -> bool:\n    p = urlparse(str(url))\n    return p.scheme in (\"http\", \"https\") and bool(p.hostname)","tryCatchPattern":"try:\n    validate_webhook_url(url)\nexcept ValueError as e:\n    if \"hostname\" in str(e):\n        url = f\"https://{url}\"  # repair scheme-less input, then revalidate\n        validate_webhook_url(url)","preventionTips":["Validate webhook URLs once at config load, not per request","Normalize user input with a scheme-prefixing step before validation"],"tags":["crawl4ai","validation","webhook","url-parsing"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}