{"record":{"id":"c202afac652796b5","repo":"assafelovic/gpt-researcher","slug":"url-scheme-scheme-or-none-r-is-not-allowed","errorCode":null,"errorMessage":"URL scheme {scheme or '(none)'!r} is not allowed; only http and https URLs may be fetched.","messagePattern":"URL scheme (.+?) is not allowed; only http and https URLs may be fetched\\.","errorType":"validation","errorClass":"UnsafeURLError","httpStatus":null,"severity":"warning","filePath":"gpt_researcher/utils/url_security.py","lineNumber":83,"sourceCode":"        allow_private: When ``True``, skip the private/internal address check.\n            When ``None`` (default), fall back to the ``ALLOW_PRIVATE_URLS``\n            environment variable.\n\n    Returns:\n        The original ``url`` if it passes all checks.\n\n    Raises:\n        UnsafeURLError: If the URL uses a disallowed scheme, lacks a host, or\n            resolves to a non-public address.\n    \"\"\"\n    if not isinstance(url, str) or not url.strip():\n        raise UnsafeURLError(\"URL must be a non-empty string.\")\n\n    parsed = urlparse(url.strip())\n\n    scheme = parsed.scheme.lower()\n    if scheme not in ALLOWED_SCHEMES:\n        raise UnsafeURLError(\n            f\"URL scheme {scheme or '(none)'!r} is not allowed; \"\n            \"only http and https URLs may be fetched.\"\n        )\n\n    host = parsed.hostname\n    if not host:\n        raise UnsafeURLError(\"URL must include a valid host.\")\n\n    if allow_private is None:\n        allow_private = _private_urls_allowed()\n    if allow_private:\n        return url\n\n    try:\n        addrinfo = socket.getaddrinfo(host, None)\n    except socket.gaierror as exc:\n        raise UnsafeURLError(f\"Could not resolve host {host!r}: {exc}\") from exc\n","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/assafelovic/gpt-researcher/blob/6f998577d547b1e54ec662dac63583aa11e3b84b/gpt_researcher/utils/url_security.py#L65-L101","documentation":"validate_url raises UnsafeURLError when the URL's scheme is not http or https (ALLOWED_SCHEMES). This blocks fetching of file://, ftp://, javascript:, data: and scheme-less URLs as part of SSRF protection, since non-HTTP schemes can bypass network controls or read local resources.","triggerScenarios":"Passing a URL like file:///etc/passwd, ftp://host/file, data:text/html,..., javascript:..., or a bare 'example.com/page' (no scheme) to extract_data_from_url / is_safe_url / validate_url.","commonSituations":"Scraping user-supplied or LLM-generated links that omit https://, markdown links with mailto: targets, file paths accidentally passed as URLs, or protocol-relative URLs ('//host/path') that urlparse leaves scheme-less.","solutions":["Normalize URLs before scraping: prepend 'https://' when no scheme is present","Reject or rewrite non-http(s) schemes from extracted links (drop mailto:, javascript:, ftp:)","Strip protocol-relative '//host' forms to 'https://host'","Catch UnsafeURLError per-URL and skip unsafe links instead of failing the batch"],"exampleFix":"# before\nurl = 'example.com/docs'  # UnsafeURLError: URL scheme '(none)' is not allowed\nawait scraper.extract_data_from_url(url)\n\n# after\nfrom urllib.parse import urlparse\nurl = 'example.com/docs'\nif not urlparse(url).scheme:\n    url = 'https://' + url\nawait scraper.extract_data_from_url(url)","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\ndef normalize_url(u: str) -> str | None:\n    u = u.strip()\n    if not u:\n        return None\n    if not urlparse(u).scheme:\n        u = 'https://' + u.lstrip('/')\n    if urlparse(u).scheme not in ('http', 'https'):\n        return None\n    return u","typeGuard":"def is_fetchable_url(u) -> bool:\n    try:\n        p = urlparse(u.strip())\n        return p.scheme in ('http', 'https') and bool(p.hostname)\n    except Exception:\n        return False","tryCatchPattern":"from gpt_researcher.utils.url_security import UnsafeURLError\n\ntry:\n    content = await scraper.extract_data_from_url(url)\nexcept UnsafeURLError as e:\n    logger.info('Skipped unsafe URL %r: %s', url, e)\n    content = ''","preventionTips":["Normalize scheme-less and protocol-relative links to https:// before scraping","Strip mailto:, javascript:, and ftp: links from scraped/LLM-generated link sets","Catch UnsafeURLError per-URL and continue; it's a skip condition, not a crash"],"tags":["ssrf-protection","url-scheme","validation","security"],"backgroundTag":"url-scheme-not-allowed","analyzedSha":"6f998577d547b1e54ec662dac63583aa11e3b84b","analyzedAt":"2026-08-28T17:50:07.383Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}