assafelovic/gpt-researcher · error · UnsafeURLError

URL must include a valid host.

Error message

URL must include a valid host.

What it means

Raised by validate_url when a parsed URL has no hostname component (e.g. 'http:///path' or a relative URL). gpt-researcher validates every URL it fetches to block SSRF, and a missing host makes DNS-level checks impossible, so the fetch is refused before any network call.

Source

Thrown at gpt_researcher/utils/url_security.py:90

    Raises:
        UnsafeURLError: If the URL uses a disallowed scheme, lacks a host, or
            resolves to a non-public address.
    """
    if not isinstance(url, str) or not url.strip():
        raise UnsafeURLError("URL must be a non-empty string.")

    parsed = urlparse(url.strip())

    scheme = parsed.scheme.lower()
    if scheme not in ALLOWED_SCHEMES:
        raise UnsafeURLError(
            f"URL scheme {scheme or '(none)'!r} is not allowed; "
            "only http and https URLs may be fetched."
        )

    host = parsed.hostname
    if not host:
        raise UnsafeURLError("URL must include a valid host.")

    if allow_private is None:
        allow_private = _private_urls_allowed()
    if allow_private:
        return url

    try:
        addrinfo = socket.getaddrinfo(host, None)
    except socket.gaierror as exc:
        raise UnsafeURLError(f"Could not resolve host {host!r}: {exc}") from exc

    for info in addrinfo:
        ip_str = info[4][0]
        try:
            ip = ipaddress.ip_address(ip_str)
        except ValueError as exc:
            raise UnsafeURLError(
                f"Host {host!r} resolved to an invalid address {ip_str!r}."

View on GitHub (pinned to 6f998577d5)

Solutions

1. Fix the URL to include a hostname, e.g. 'https://example.com/page'.
2. Validate/normalize URLs with urllib.parse.urlparse and check .hostname before passing them to gpt-researcher.
3. Catch UnsafeURLError at the call site and skip/log the malformed source.

Example fix

// before
await researcher.extract_data_from_url("/docs/intro")
// after
await researcher.extract_data_from_url("https://example.com/docs/intro")

from urllib.parse import urlparse

def has_valid_host(url: str) -> bool:
    return urlparse(url).scheme in ("http", "https") and bool(urlparse(url).hostname)

from gpt_researcher.utils.url_security import UnsafeURLError
try:
    validate_url(url)
except UnsafeURLError as e:
    logger.warning("skipping unsafe/invalid url %s: %s", url, e)

When it happens

Trigger: Calling validate_url, extract_data_from_url, is_safe_url, or the scraper's _download_and_process with a URL like 'file:///etc/passwd' variant that passes scheme checks but parses without a hostname, 'http:///foo', or an empty/malformed string.

Common situations: Passing a relative path instead of a full URL, copy-pasting URLs with a missing domain, or constructing URLs by string concatenation that drops the host.

Related errors

• Could not resolve host {host!r}: {exc}
• Host {host!r} resolved to an invalid address {ip_str!r}.
• URL host {host!r} resolves to a non-public address ({ip_str}
• Cost must be an integer or float
• Embedding provider not found.

AI-assisted analysis of assafelovic/gpt-researcher@6f998577d5 (2026-08-28). Data as JSON: /api/errors/6de7e516d72caaf3. Report an issue: GitHub.