assafelovic/gpt-researcher · error · UnsafeURLError

URL host {host!r} resolves to a non-public address ({ip_str}

Error message

URL host {host!r} resolves to a non-public address ({ip_str}); set ALLOW_PRIVATE_URLS=true to allow internal targets.

What it means

validate_url resolves the hostname and rejects any address that is private, loopback, link-local, or otherwise non-public, to prevent SSRF against internal services. Setting ALLOW_PRIVATE_URLS=true (or passing allow_private=True) bypasses the check.

Source

Thrown at gpt_researcher/utils/url_security.py:111

        allow_private = _private_urls_allowed()
    if allow_private:
        return url

    try:
        addrinfo = socket.getaddrinfo(host, None)
    except socket.gaierror as exc:
        raise UnsafeURLError(f"Could not resolve host {host!r}: {exc}") from exc

    for info in addrinfo:
        ip_str = info[4][0]
        try:
            ip = ipaddress.ip_address(ip_str)
        except ValueError as exc:
            raise UnsafeURLError(
                f"Host {host!r} resolved to an invalid address {ip_str!r}."
            ) from exc
        if _is_disallowed_ip(ip):
            raise UnsafeURLError(
                f"URL host {host!r} resolves to a non-public address ({ip_str}); "
                "set ALLOW_PRIVATE_URLS=true to allow internal targets."
            )

    return url


def is_safe_url(url: str, *, allow_private: bool | None = None) -> bool:
    """Return ``True`` if ``url`` passes :func:`validate_url`, else ``False``."""
    try:
        validate_url(url, allow_private=allow_private)
        return True
    except UnsafeURLError:
        return False

View on GitHub (pinned to 6f998577d5)

Solutions

Set env var ALLOW_PRIVATE_URLS=true before creating the researcher (only in trusted environments — it disables SSRF protection).
Expose the local service on a public hostname/IP and fetch that instead.
Pass allow_private=True programmatically to validate_url if calling it directly.

Example fix

# before
researcher = GPTResearcher(query="...", report_source="web")  # fetching localhost fails
# after
import os
os.environ["ALLOW_PRIVATE_URLS"] = "true"
researcher = GPTResearcher(query="...", report_source="web")

Defensive patterns

Strategy: fallback

Validate before calling

import ipaddress, socket
from urllib.parse import urlparse

def is_public(url: str) -> bool:
    host = urlparse(url).hostname
    if not host:
        return False
    try:
        return all(not ipaddress.ip_address(i[4][0]).is_private for i in socket.getaddrinfo(host, None))
    except socket.gaierror:
        return False

Try / catch

from gpt_researcher.utils.url_security import UnsafeURLError
try:
    validate_url(url)
except UnsafeURLError as e:
    if "non-public address" in str(e) and TRUST_INTERNAL:
        os.environ["ALLOW_PRIVATE_URLS"] = "true"  # then retry deliberately
    else:
        raise

Prevention

Set ALLOW_PRIVATE_URLS=true only in trusted, isolated environments.
Prefer exposing internal services via public hostnames rather than disabling the guard.
Never let end-user input pick URLs without this validation.

When it happens

Trigger: Fetching http://localhost:8000, http://127.0.0.1, http://192.168.x.x, http://10.x.x.x, 169.254.169.254 (cloud metadata), or a public name that DNS-resolves to a private IP, without ALLOW_PRIVATE_URLS set.

Common situations: Developers testing against a local dev server or internal service; deployments behind NAT where a public hostname resolves internally; intentional intranet crawling in Docker/Kubernetes.

Related errors

AI-assisted analysis of assafelovic/gpt-researcher@6f998577d5 (2026-08-28). Data as JSON: /api/errors/30e19621ab9e72d7. Report an issue: GitHub.