assafelovic/gpt-researcher · warning · UnsafeURLError

URL must be a non-empty string.

Error message

URL must be a non-empty string.

What it means

validate_url in gpt_researcher.utils.url_security raises UnsafeURLError when the url argument is not a string or is empty/whitespace-only. This is the first check in the SSRF-protection pipeline that all fetched URLs must pass before any network request is made.

Source

Thrown at gpt_researcher/utils/url_security.py:77

def validate_url(url: str, *, allow_private: bool | None = None) -> str:
    """Validate that ``url`` is safe to fetch and return it unchanged.

    Args:
        url: The URL to validate.
        allow_private: When ``True``, skip the private/internal address check.
            When ``None`` (default), fall back to the ``ALLOW_PRIVATE_URLS``
            environment variable.

    Returns:
        The original ``url`` if it passes all checks.

    Raises:
        UnsafeURLError: If the URL uses a disallowed scheme, lacks a host, or
            resolves to a non-public address.
    """
    if not isinstance(url, str) or not url.strip():
        raise UnsafeURLError("URL must be a non-empty string.")

    parsed = urlparse(url.strip())

    scheme = parsed.scheme.lower()
    if scheme not in ALLOWED_SCHEMES:
        raise UnsafeURLError(
            f"URL scheme {scheme or '(none)'!r} is not allowed; "
            "only http and https URLs may be fetched."
        )

    host = parsed.hostname
    if not host:
        raise UnsafeURLError("URL must include a valid host.")

    if allow_private is None:
        allow_private = _private_urls_allowed()
    if allow_private:
        return url

View on GitHub (pinned to 6f998577d5)

Solutions

1. Filter out falsy/None URLs before scraping: if not url or not url.strip(): skip
2. Guard with isinstance(url, str) when URLs come from external data
3. Catch UnsafeURLError around scrape calls and skip the bad URL rather than aborting the run
4. Log the offending value to find the upstream source of empty URLs

Example fix

# before
content = await scraper.extract_data_from_url(result.get('url'))  # None -> UnsafeURLError

# after
url = result.get('url')
if isinstance(url, str) and url.strip():
    content = await scraper.extract_data_from_url(url)

urls = [u for u in candidates if isinstance(u, str) and u.strip()]
if not urls:
    return  # nothing safe to fetch

def is_url_string(v) -> bool:
    return isinstance(v, str) and len(v.strip()) > 0

from gpt_researcher.utils.url_security import UnsafeURLError

try:
    await scraper.extract_data_from_url(url)
except UnsafeURLError:
    continue  # skip bad URL, keep processing the batch

When it happens

Trigger: Calling extract_data_from_url, is_safe_url, or validate_url with None, an empty string, a bytes URL, or a whitespace string; often the result of upstream parsing that produced no URL (e.g. a search result with a missing href).

Common situations: Feeding uncleaned search/SERP results into the scraper, None slipping through after a failed lookup, list/dict passed where a URL string was expected, or whitespace-only strings from trimmed config.

Related errors

• Model cannot be None
• URL scheme {scheme or '(none)'!r} is not allowed; only http
• Invalid retriever(s) found: {', '.join(invalid_retrievers)}.
• Invalid reasoning effort: {reasoning_effort_str}. Valid opti
• Scraper not found.

AI-assisted analysis of assafelovic/gpt-researcher@6f998577d5 (2026-08-28). Data as JSON: /api/errors/98249e39749498f8. Report an issue: GitHub.