oobabooga/textgen · error · ValueError

Unsupported URL scheme: {parsed.scheme}

Error message

Unsupported URL scheme: {parsed.scheme}

What it means

Raised by _validate_url() when urlparse() extracts a scheme other than http or https. The web-fetch layer only supports cleartext and TLS HTTP; schemes like ftp, file, gopher, or an empty scheme (relative URL) are rejected before any DNS resolution, both for functionality and to block file:// and similar SSRF vectors.

Source

Thrown at modules/web_search.py:24

from urllib.parse import urljoin, urlparse

import requests
from ddgs import DDGS

from modules import shared
from modules.logging_colors import logger


def _validate_url(url):
    """Validate that a URL is safe to fetch (not targeting private/internal networks)."""
    # Reject characters that cause parsing discrepancies between urlparse and requests,
    # which can be exploited to bypass SSRF protections (GHSA-27xf-58m5-vxmc).
    if '\\' in url:
        raise ValueError("Invalid URL: backslashes are not allowed")

    parsed = urlparse(url)
    if parsed.scheme not in ('http', 'https'):
        raise ValueError(f"Unsupported URL scheme: {parsed.scheme}")

    if '@' in parsed.netloc:
        raise ValueError("Invalid URL: userinfo (credentials) in URLs is not allowed")

    hostname = parsed.hostname
    if not hostname:
        raise ValueError("No hostname in URL")

    # Resolve hostname and check all returned addresses
    try:
        for family, _, _, _, sockaddr in socket.getaddrinfo(hostname, None):
            ip = ipaddress.ip_address(sockaddr[0])
            if not ip.is_global:
                raise ValueError(f"Access to non-public address {ip} is blocked")
    except socket.gaierror:
        raise ValueError(f"Could not resolve hostname: {hostname}")

View on GitHub (pinned to ed888c71f2)

Solutions

1. Prefix the URL with 'https://' when the user supplied only a hostname.
2. Strip or replace non-http links (mailto:, tel:, ftp:) from scraped/search result lists before fetching.
3. If you control the redirect source, ensure Location headers always use http/https absolute or path-relative URLs.
4. Never pass file:// or other local schemes — the guard intentionally blocks them.

Example fix

# before
resp = safe_get('example.com/page')  # parsed.scheme == '' -> raises

# after
url = 'example.com/page'
if '://' not in url:
    url = 'https://' + url
resp = safe_get(url)

from urllib.parse import urlparse

def is_http_url(url: str) -> bool:
    try:
        return urlparse(url).scheme in ('http', 'https') and bool(urlparse(url).hostname)
    except Exception:
        return False

url = url if '://' in url else 'https://' + url
if not is_http_url(url):
    raise ValueError(f'Rejecting non-http URL: {url!r}')

try:
    resp = safe_get(url)
except ValueError as e:
    if 'Unsupported URL scheme' in str(e):
        log.warning('Skipped non-http link: %s', url)  # e.g. mailto:, ftp: from scraped lists
    else:
        raise

When it happens

Trigger: Passing a URL like 'ftp://example.com/file', 'file:///etc/passwd', 'javascript:...', or a scheme-less string like 'example.com/page' to safe_get()/download_web_page(). Also triggered mid-redirect if a server returns a Location header with a non-http scheme (e.g. an ftp:// or protocol-relative malformed value).

Common situations: User submits a bare domain without 'https://' in the web-search UI; feed or search results contain non-http links; a redirect target switches schemes; attempts to fetch local files through the web-fetch endpoint.

Related errors

• Invalid URL: backslashes are not allowed
• No hostname in URL
• Invalid URL: userinfo (credentials) in URLs is not allowed
• Access to non-public address {ip} is blocked
• Could not resolve hostname: {hostname}

AI-assisted analysis of oobabooga/textgen@ed888c71f2 (2026-08-15). Data as JSON: /api/errors/f47c252f55f4eefa. Report an issue: GitHub.