assafelovic/gpt-researcher · warning · UnsafeURLError

URL scheme {scheme or '(none)'!r} is not allowed; only http

Error message

URL scheme {scheme or '(none)'!r} is not allowed; only http and https URLs may be fetched.

What it means

validate_url raises UnsafeURLError when the URL's scheme is not http or https (ALLOWED_SCHEMES). This blocks fetching of file://, ftp://, javascript:, data: and scheme-less URLs as part of SSRF protection, since non-HTTP schemes can bypass network controls or read local resources.

Source

Thrown at gpt_researcher/utils/url_security.py:83

        allow_private: When ``True``, skip the private/internal address check.
            When ``None`` (default), fall back to the ``ALLOW_PRIVATE_URLS``
            environment variable.

    Returns:
        The original ``url`` if it passes all checks.

    Raises:
        UnsafeURLError: If the URL uses a disallowed scheme, lacks a host, or
            resolves to a non-public address.
    """
    if not isinstance(url, str) or not url.strip():
        raise UnsafeURLError("URL must be a non-empty string.")

    parsed = urlparse(url.strip())

    scheme = parsed.scheme.lower()
    if scheme not in ALLOWED_SCHEMES:
        raise UnsafeURLError(
            f"URL scheme {scheme or '(none)'!r} is not allowed; "
            "only http and https URLs may be fetched."
        )

    host = parsed.hostname
    if not host:
        raise UnsafeURLError("URL must include a valid host.")

    if allow_private is None:
        allow_private = _private_urls_allowed()
    if allow_private:
        return url

    try:
        addrinfo = socket.getaddrinfo(host, None)
    except socket.gaierror as exc:
        raise UnsafeURLError(f"Could not resolve host {host!r}: {exc}") from exc

View on GitHub (pinned to 6f998577d5)

Solutions

1. Normalize URLs before scraping: prepend 'https://' when no scheme is present
2. Reject or rewrite non-http(s) schemes from extracted links (drop mailto:, javascript:, ftp:)
3. Strip protocol-relative '//host' forms to 'https://host'
4. Catch UnsafeURLError per-URL and skip unsafe links instead of failing the batch

Example fix

# before
url = 'example.com/docs'  # UnsafeURLError: URL scheme '(none)' is not allowed
await scraper.extract_data_from_url(url)

# after
from urllib.parse import urlparse
url = 'example.com/docs'
if not urlparse(url).scheme:
    url = 'https://' + url
await scraper.extract_data_from_url(url)

from urllib.parse import urlparse

def normalize_url(u: str) -> str | None:
    u = u.strip()
    if not u:
        return None
    if not urlparse(u).scheme:
        u = 'https://' + u.lstrip('/')
    if urlparse(u).scheme not in ('http', 'https'):
        return None
    return u

def is_fetchable_url(u) -> bool:
    try:
        p = urlparse(u.strip())
        return p.scheme in ('http', 'https') and bool(p.hostname)
    except Exception:
        return False

from gpt_researcher.utils.url_security import UnsafeURLError

try:
    content = await scraper.extract_data_from_url(url)
except UnsafeURLError as e:
    logger.info('Skipped unsafe URL %r: %s', url, e)
    content = ''

When it happens

Trigger: Passing a URL like file:///etc/passwd, ftp://host/file, data:text/html,..., javascript:..., or a bare 'example.com/page' (no scheme) to extract_data_from_url / is_safe_url / validate_url.

Common situations: Scraping user-supplied or LLM-generated links that omit https://, markdown links with mailto: targets, file paths accidentally passed as URLs, or protocol-relative URLs ('//host/path') that urlparse leaves scheme-less.

Related errors

• URL must be a non-empty string.
• URL host {host!r} resolves to a non-public address ({ip_str}
• Invalid retriever(s) found: {', '.join(invalid_retrievers)}.
• Invalid reasoning effort: {reasoning_effort_str}. Valid opti
• Unsafe blob name: {blob_name}

AI-assisted analysis of assafelovic/gpt-researcher@6f998577d5 (2026-08-28). Data as JSON: /api/errors/c202afac652796b5. Report an issue: GitHub.