{"record":{"id":"98249e39749498f8","repo":"assafelovic/gpt-researcher","slug":"url-must-be-a-non-empty-string","errorCode":null,"errorMessage":"URL must be a non-empty string.","messagePattern":"URL must be a non-empty string\\.","errorType":"validation","errorClass":"UnsafeURLError","httpStatus":null,"severity":"warning","filePath":"gpt_researcher/utils/url_security.py","lineNumber":77,"sourceCode":"\ndef validate_url(url: str, *, allow_private: bool | None = None) -> str:\n    \"\"\"Validate that ``url`` is safe to fetch and return it unchanged.\n\n    Args:\n        url: The URL to validate.\n        allow_private: When ``True``, skip the private/internal address check.\n            When ``None`` (default), fall back to the ``ALLOW_PRIVATE_URLS``\n            environment variable.\n\n    Returns:\n        The original ``url`` if it passes all checks.\n\n    Raises:\n        UnsafeURLError: If the URL uses a disallowed scheme, lacks a host, or\n            resolves to a non-public address.\n    \"\"\"\n    if not isinstance(url, str) or not url.strip():\n        raise UnsafeURLError(\"URL must be a non-empty string.\")\n\n    parsed = urlparse(url.strip())\n\n    scheme = parsed.scheme.lower()\n    if scheme not in ALLOWED_SCHEMES:\n        raise UnsafeURLError(\n            f\"URL scheme {scheme or '(none)'!r} is not allowed; \"\n            \"only http and https URLs may be fetched.\"\n        )\n\n    host = parsed.hostname\n    if not host:\n        raise UnsafeURLError(\"URL must include a valid host.\")\n\n    if allow_private is None:\n        allow_private = _private_urls_allowed()\n    if allow_private:\n        return url","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/assafelovic/gpt-researcher/blob/6f998577d547b1e54ec662dac63583aa11e3b84b/gpt_researcher/utils/url_security.py#L59-L95","documentation":"validate_url in gpt_researcher.utils.url_security raises UnsafeURLError when the url argument is not a string or is empty/whitespace-only. This is the first check in the SSRF-protection pipeline that all fetched URLs must pass before any network request is made.","triggerScenarios":"Calling extract_data_from_url, is_safe_url, or validate_url with None, an empty string, a bytes URL, or a whitespace string; often the result of upstream parsing that produced no URL (e.g. a search result with a missing href).","commonSituations":"Feeding uncleaned search/SERP results into the scraper, None slipping through after a failed lookup, list/dict passed where a URL string was expected, or whitespace-only strings from trimmed config.","solutions":["Filter out falsy/None URLs before scraping: if not url or not url.strip(): skip","Guard with isinstance(url, str) when URLs come from external data","Catch UnsafeURLError around scrape calls and skip the bad URL rather than aborting the run","Log the offending value to find the upstream source of empty URLs"],"exampleFix":"# before\ncontent = await scraper.extract_data_from_url(result.get('url'))  # None -> UnsafeURLError\n\n# after\nurl = result.get('url')\nif isinstance(url, str) and url.strip():\n    content = await scraper.extract_data_from_url(url)","handlingStrategy":"type-guard","validationCode":"urls = [u for u in candidates if isinstance(u, str) and u.strip()]\nif not urls:\n    return  # nothing safe to fetch","typeGuard":"def is_url_string(v) -> bool:\n    return isinstance(v, str) and len(v.strip()) > 0","tryCatchPattern":"from gpt_researcher.utils.url_security import UnsafeURLError\n\ntry:\n    await scraper.extract_data_from_url(url)\nexcept UnsafeURLError:\n    continue  # skip bad URL, keep processing the batch","preventionTips":["Treat external/LLM-generated link lists as untrusted: filter before fetching","Default-None dictionary gets (.get('url')) to a safe skip, not a scrape call","Catch UnsafeURLError per-URL so one bad link doesn't kill a research run"],"tags":["validation","ssrf-protection","url","invalid-argument"],"backgroundTag":"url-validation-failed","analyzedSha":"6f998577d547b1e54ec662dac63583aa11e3b84b","analyzedAt":"2026-08-28T17:50:07.383Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}