{"record":{"id":"c4c3c84197185ff7","repo":"assafelovic/gpt-researcher","slug":"could-not-resolve-host-host-r-exc","errorCode":null,"errorMessage":"Could not resolve host {host!r}: {exc}","messagePattern":"Could not resolve host (.+?): (.+?)","errorType":"validation","errorClass":"UnsafeURLError","httpStatus":null,"severity":"error","filePath":"gpt_researcher/utils/url_security.py","lineNumber":100,"sourceCode":"    if scheme not in ALLOWED_SCHEMES:\n        raise UnsafeURLError(\n            f\"URL scheme {scheme or '(none)'!r} is not allowed; \"\n            \"only http and https URLs may be fetched.\"\n        )\n\n    host = parsed.hostname\n    if not host:\n        raise UnsafeURLError(\"URL must include a valid host.\")\n\n    if allow_private is None:\n        allow_private = _private_urls_allowed()\n    if allow_private:\n        return url\n\n    try:\n        addrinfo = socket.getaddrinfo(host, None)\n    except socket.gaierror as exc:\n        raise UnsafeURLError(f\"Could not resolve host {host!r}: {exc}\") from exc\n\n    for info in addrinfo:\n        ip_str = info[4][0]\n        try:\n            ip = ipaddress.ip_address(ip_str)\n        except ValueError as exc:\n            raise UnsafeURLError(\n                f\"Host {host!r} resolved to an invalid address {ip_str!r}.\"\n            ) from exc\n        if _is_disallowed_ip(ip):\n            raise UnsafeURLError(\n                f\"URL host {host!r} resolves to a non-public address ({ip_str}); \"\n                \"set ALLOW_PRIVATE_URLS=true to allow internal targets.\"\n            )\n\n    return url\n\n","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/assafelovic/gpt-researcher/blob/6f998577d547b1e54ec662dac63583aa11e3b84b/gpt_researcher/utils/url_security.py#L82-L118","documentation":"Raised when validate_url cannot DNS-resolve the URL's hostname (socket.getaddrinfo raises gaierror). Because gpt-researcher must resolve the host to verify it points at a public IP (SSRF mitigation), an unresolvable host aborts the fetch.","triggerScenarios":"Calling extract_data_from_url / _download_and_process / validate_url with a hostname that fails DNS resolution: typo'd domain, internal-only DNS name while ALLOW_PRIVATE_URLS is unset, or a sandboxed environment with no network/DNS access.","commonSituations":"CI containers with restricted DNS, air-gapped test environments (tests mock this with patching), typo'd domains, or '.local'/.internal hostnames in cluster deployments.","solutions":["Verify the hostname resolves: run 'nslookup <host>' or 'socket.getaddrinfo(host, None)' in the same environment.","If the target is an internal host you trust, set ALLOW_PRIVATE_URLS=true to skip resolution/public-IP checks.","Fix typos in the URL or use an IP/public DNS-resolvable name.","In tests, mock socket.getaddrinfo or gpt_researcher.utils.url_security.validate_url."],"exampleFix":"# before\nawait researcher.extract_data_from_url(\"https://internal-svc.local/doc\")\n# after (trusted internal target)\nos.environ[\"ALLOW_PRIVATE_URLS\"] = \"true\"\nawait researcher.extract_data_from_url(\"https://internal-svc.local/doc\")","handlingStrategy":"try-catch","validationCode":"import socket\n\ndef resolvable(host: str) -> bool:\n    try:\n        socket.getaddrinfo(host, None)\n        return True\n    except socket.gaierror:\n        return False","typeGuard":null,"tryCatchPattern":"from gpt_researcher.utils.url_security import UnsafeURLError\ntry:\n    validate_url(url)\nexcept UnsafeURLError as e:\n    if \"Could not resolve\" in str(e):\n        logger.warning(\"DNS failure for %s; skipping\", url)\n    else:\n        raise","preventionTips":["Pre-check DNS in the same network namespace the researcher runs in.","Set ALLOW_PRIVATE_URLS=true for trusted internal targets.","Mock getaddrinfo/validate_url in unit tests instead of relying on live DNS."],"tags":["dns","network","ssrf-protection","python"],"backgroundTag":"dns-resolution-failed","analyzedSha":"6f998577d547b1e54ec662dac63583aa11e3b84b","analyzedAt":"2026-08-28T17:50:07.383Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}