{"record":{"id":"f47c252f55f4eefa","repo":"oobabooga/textgen","slug":"unsupported-url-scheme-parsed-scheme","errorCode":null,"errorMessage":"Unsupported URL scheme: {parsed.scheme}","messagePattern":"Unsupported URL scheme: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"modules/web_search.py","lineNumber":24,"sourceCode":"from urllib.parse import urljoin, urlparse\n\nimport requests\nfrom ddgs import DDGS\n\nfrom modules import shared\nfrom modules.logging_colors import logger\n\n\ndef _validate_url(url):\n    \"\"\"Validate that a URL is safe to fetch (not targeting private/internal networks).\"\"\"\n    # Reject characters that cause parsing discrepancies between urlparse and requests,\n    # which can be exploited to bypass SSRF protections (GHSA-27xf-58m5-vxmc).\n    if '\\\\' in url:\n        raise ValueError(\"Invalid URL: backslashes are not allowed\")\n\n    parsed = urlparse(url)\n    if parsed.scheme not in ('http', 'https'):\n        raise ValueError(f\"Unsupported URL scheme: {parsed.scheme}\")\n\n    if '@' in parsed.netloc:\n        raise ValueError(\"Invalid URL: userinfo (credentials) in URLs is not allowed\")\n\n    hostname = parsed.hostname\n    if not hostname:\n        raise ValueError(\"No hostname in URL\")\n\n    # Resolve hostname and check all returned addresses\n    try:\n        for family, _, _, _, sockaddr in socket.getaddrinfo(hostname, None):\n            ip = ipaddress.ip_address(sockaddr[0])\n            if not ip.is_global:\n                raise ValueError(f\"Access to non-public address {ip} is blocked\")\n    except socket.gaierror:\n        raise ValueError(f\"Could not resolve hostname: {hostname}\")\n\n","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/oobabooga/textgen/blob/ed888c71f221df552750e1834b3654abab8ae345/modules/web_search.py#L6-L42","documentation":"Raised by _validate_url() when urlparse() extracts a scheme other than http or https. The web-fetch layer only supports cleartext and TLS HTTP; schemes like ftp, file, gopher, or an empty scheme (relative URL) are rejected before any DNS resolution, both for functionality and to block file:// and similar SSRF vectors.","triggerScenarios":"Passing a URL like 'ftp://example.com/file', 'file:///etc/passwd', 'javascript:...', or a scheme-less string like 'example.com/page' to safe_get()/download_web_page(). Also triggered mid-redirect if a server returns a Location header with a non-http scheme (e.g. an ftp:// or protocol-relative malformed value).","commonSituations":"User submits a bare domain without 'https://' in the web-search UI; feed or search results contain non-http links; a redirect target switches schemes; attempts to fetch local files through the web-fetch endpoint.","solutions":["Prefix the URL with 'https://' when the user supplied only a hostname.","Strip or replace non-http links (mailto:, tel:, ftp:) from scraped/search result lists before fetching.","If you control the redirect source, ensure Location headers always use http/https absolute or path-relative URLs.","Never pass file:// or other local schemes — the guard intentionally blocks them."],"exampleFix":"# before\nresp = safe_get('example.com/page')  # parsed.scheme == '' -> raises\n\n# after\nurl = 'example.com/page'\nif '://' not in url:\n    url = 'https://' + url\nresp = safe_get(url)","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\n\ndef is_http_url(url: str) -> bool:\n    try:\n        return urlparse(url).scheme in ('http', 'https') and bool(urlparse(url).hostname)\n    except Exception:\n        return False\n\nurl = url if '://' in url else 'https://' + url\nif not is_http_url(url):\n    raise ValueError(f'Rejecting non-http URL: {url!r}')","typeGuard":null,"tryCatchPattern":"try:\n    resp = safe_get(url)\nexcept ValueError as e:\n    if 'Unsupported URL scheme' in str(e):\n        log.warning('Skipped non-http link: %s', url)  # e.g. mailto:, ftp: from scraped lists\n    else:\n        raise","preventionTips":["Always prefix bare hostnames with https:// before fetching.","Filter scraped/search result links to http(s) before fetching them in bulk.","Remember file:// and ftp:// are intentionally blocked by the SSRF guard — fetch local files directly instead."],"tags":["ssrf","url-validation","security","web-fetch"],"backgroundTag":null,"analyzedSha":"ed888c71f221df552750e1834b3654abab8ae345","analyzedAt":"2026-08-15T05:24:21.000Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}