{"record":{"id":"6de7e516d72caaf3","repo":"assafelovic/gpt-researcher","slug":"url-must-include-a-valid-host","errorCode":null,"errorMessage":"URL must include a valid host.","messagePattern":"URL must include a valid host\\.","errorType":"validation","errorClass":"UnsafeURLError","httpStatus":null,"severity":"error","filePath":"gpt_researcher/utils/url_security.py","lineNumber":90,"sourceCode":"    Raises:\n        UnsafeURLError: If the URL uses a disallowed scheme, lacks a host, or\n            resolves to a non-public address.\n    \"\"\"\n    if not isinstance(url, str) or not url.strip():\n        raise UnsafeURLError(\"URL must be a non-empty string.\")\n\n    parsed = urlparse(url.strip())\n\n    scheme = parsed.scheme.lower()\n    if scheme not in ALLOWED_SCHEMES:\n        raise UnsafeURLError(\n            f\"URL scheme {scheme or '(none)'!r} is not allowed; \"\n            \"only http and https URLs may be fetched.\"\n        )\n\n    host = parsed.hostname\n    if not host:\n        raise UnsafeURLError(\"URL must include a valid host.\")\n\n    if allow_private is None:\n        allow_private = _private_urls_allowed()\n    if allow_private:\n        return url\n\n    try:\n        addrinfo = socket.getaddrinfo(host, None)\n    except socket.gaierror as exc:\n        raise UnsafeURLError(f\"Could not resolve host {host!r}: {exc}\") from exc\n\n    for info in addrinfo:\n        ip_str = info[4][0]\n        try:\n            ip = ipaddress.ip_address(ip_str)\n        except ValueError as exc:\n            raise UnsafeURLError(\n                f\"Host {host!r} resolved to an invalid address {ip_str!r}.\"","sourceCodeStart":72,"sourceCodeEnd":108,"githubUrl":"https://github.com/assafelovic/gpt-researcher/blob/6f998577d547b1e54ec662dac63583aa11e3b84b/gpt_researcher/utils/url_security.py#L72-L108","documentation":"Raised by validate_url when a parsed URL has no hostname component (e.g. 'http:///path' or a relative URL). gpt-researcher validates every URL it fetches to block SSRF, and a missing host makes DNS-level checks impossible, so the fetch is refused before any network call.","triggerScenarios":"Calling validate_url, extract_data_from_url, is_safe_url, or the scraper's _download_and_process with a URL like 'file:///etc/passwd' variant that passes scheme checks but parses without a hostname, 'http:///foo', or an empty/malformed string.","commonSituations":"Passing a relative path instead of a full URL, copy-pasting URLs with a missing domain, or constructing URLs by string concatenation that drops the host.","solutions":["Fix the URL to include a hostname, e.g. 'https://example.com/page'.","Validate/normalize URLs with urllib.parse.urlparse and check .hostname before passing them to gpt-researcher.","Catch UnsafeURLError at the call site and skip/log the malformed source."],"exampleFix":"// before\nawait researcher.extract_data_from_url(\"/docs/intro\")\n// after\nawait researcher.extract_data_from_url(\"https://example.com/docs/intro\")","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\ndef has_valid_host(url: str) -> bool:\n    return urlparse(url).scheme in (\"http\", \"https\") and bool(urlparse(url).hostname)","typeGuard":null,"tryCatchPattern":"from gpt_researcher.utils.url_security import UnsafeURLError\ntry:\n    validate_url(url)\nexcept UnsafeURLError as e:\n    logger.warning(\"skipping unsafe/invalid url %s: %s\", url, e)","preventionTips":["Always build URLs from a base with urljoin so the host is never lost.","Reject non-http(s) schemes client-side before handing URLs to the researcher."],"tags":["url-validation","ssrf-protection","python"],"backgroundTag":"invalid-url-format","analyzedSha":"6f998577d547b1e54ec662dac63583aa11e3b84b","analyzedAt":"2026-08-28T17:50:07.383Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}