oobabooga/textgen · warning · ValueError

Too many redirects (max {max_redirects})

Error message

Too many redirects (max {max_redirects})

What it means

Raised by safe_get() when the redirect chain exceeds max_redirects (default 5) hops without a terminal response. The function follows redirects manually (allow_redirects=False) so that every hop can be re-validated by _validate_url(); once the budget is exhausted it refuses to continue rather than looping indefinitely, protecting against redirect loops and chains used to exhaust resources.

Source

Thrown at modules/web_search.py:54

            ip = ipaddress.ip_address(sockaddr[0])
            if not ip.is_global:
                raise ValueError(f"Access to non-public address {ip} is blocked")
    except socket.gaierror:
        raise ValueError(f"Could not resolve hostname: {hostname}")


def safe_get(url, headers=None, timeout=10, max_redirects=5):
    """Fetch a URL with SSRF-safe redirect handling. Validates every hop."""
    _validate_url(url)
    for _ in range(max_redirects):
        response = requests.get(url, headers=headers, timeout=timeout, allow_redirects=False)
        if response.is_redirect and 'Location' in response.headers:
            url = urljoin(url, response.headers['Location'])
            _validate_url(url)
        else:
            return response

    raise ValueError(f"Too many redirects (max {max_redirects})")


def get_current_timestamp():
    """Returns the current time in 24-hour format"""
    return datetime.now().strftime('%b %d, %Y %H:%M')


def download_web_page(url, timeout=10, include_links=False):
    """
    Download a web page and extract its main content as Markdown text.
    """
    import trafilatura

    try:
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36'
        }
        response = safe_get(url, headers=headers, timeout=timeout)

View on GitHub (pinned to ed888c71f2)

Solutions

1. Fetch the final URL directly: open the link once in a browser/curl -L, take the landed URL, and pass that to safe_get.
2. Retry with a higher max_redirects if you control the call and trust the target (safe_get accepts max_redirects as a parameter).
3. If the site is redirect-looping due to missing cookies/headers, pass appropriate headers (User-Agent, cookies) to break the loop.
4. Skip the offending link in bulk crawling scenarios and report it as unreachable.

Example fix

# before
resp = safe_get(url)  # default max_redirects=5, loop site raises

# after
resp = safe_get(url, max_redirects=10)  # or first resolve the final URL out-of-band
# final = requests.head(url, allow_redirects=True).url
# resp = safe_get(final)

try:
    resp = safe_get(url)
except ValueError as e:
    if 'Too many redirects' in str(e):
        # resolve final URL out-of-band once, then fetch directly
        import requests
        final = requests.head(url, allow_redirects=True, timeout=10).url
        resp = safe_get(final)
    else:
        raise

When it happens

Trigger: Fetching a URL involved in a redirect loop (A -> B -> A), an excessively long chain (shortlink services chaining through many trackers), or a server that keeps issuing redirects (e.g. HTTP->HTTPS->auth->back) exceeding 5 hops. Each hop also costs a validation pass, so the cap bounds total work.

Common situations: Login-walled or CDN-protected sites that redirect repeatedly; broken server configs with self-redirects; tracker-laden short links; mirroring/crawling tools that hit sites with cookie-consent redirect churn.

Related errors

• Access to non-public address {ip} is blocked
• Could not resolve hostname: {hostname}
• Invalid URL: backslashes are not allowed
• Unsupported URL scheme: {parsed.scheme}
• No hostname in URL

AI-assisted analysis of oobabooga/textgen@ed888c71f2 (2026-08-15). Data as JSON: /api/errors/d3dd4b2084024f95. Report an issue: GitHub.