infiniflow/ragflow · error · ValueError

Failed to fetch {current_url!r}: {_exc}

Error message

Failed to fetch {current_url!r}: {_exc}

What it means

Raised in the crawl path of FileService web-crawling when the manual redirect-following requests.get fails with a RequestException (DNS failure, connection refused, TLS error, timeout of 10s). The URL was already SSRF-validated by _validate_url_for_crawl; this error means the HTTP fetch itself failed.

Source

Thrown at api/db/services/file_service.py:819

                # follows a server-sent redirect to an unvalidated (potentially
                # internal) host. Each hop is SSRF-checked before being followed;
                # the validated (hostname, ip) pairs are pinned via Chromium's
                # --host-resolver-rules so the browser cannot re-resolve any of them
                # through a fresh DNS query.
                current_url = url
                current_hostname, current_ip = FileService._validate_url_for_crawl(current_url)
                # Accumulate MAP rules for every hostname we encounter in the chain.
                host_pins: dict[str, str] = {current_hostname: current_ip}

                for _ in range(_MAX_CRAWL_REDIRECTS):
                    try:
                        _resp = _requests.get(
                            current_url,
                            timeout=10,
                            allow_redirects=False,
                        )
                    except _requests.RequestException as _exc:
                        raise ValueError(f"Failed to fetch {current_url!r}: {_exc}") from _exc

                    if _resp.status_code not in (301, 302, 303, 307, 308):
                        break

                    _location = _resp.headers.get("Location")
                    if not _location:
                        break

                    _next_url = _urljoin(current_url, _location)
                    _next_hostname, _next_ip = FileService._validate_url_for_crawl(_next_url)
                    host_pins[_next_hostname] = _next_ip
                    current_url = _next_url
                else:
                    raise ValueError(f"Exceeded {_MAX_CRAWL_REDIRECTS} redirects fetching {url!r}")

                # Build a single MAP rule string covering every validated hostname
                # in the redirect chain. Chromium uses the pinned IP for each,
                # skipping DNS entirely and eliminating the rebinding window.

View on GitHub (pinned to 554fb1133a)

Solutions

1. Verify the URL is reachable from the RAGFlow server container: curl -I <url>.
2. Fix DNS/network/firewall access between the server and the target host.
3. If the site is slow, the 10s timeout in this function must be raised in code (it is hard-coded).
4. Retry transient network failures; skip the URL if it is permanently unreachable.

import requests as r
try:
    probe = r.head(url, timeout=10, allow_redirects=False)
except r.RequestException:
    return json_error_response('URL unreachable from server', 400)

def is_reachable_url(url: str, timeout: float = 5.0) -> bool:
    try:
        r = _requests.get(url, timeout=timeout, allow_redirects=False)
        return r.status_code < 500
    except _requests.RequestException:
        return False

try:
    FileService.web_crawl(url)
except ValueError as e:
    if str(e).startswith('Failed to fetch'):
        return json_error_response('target URL unreachable', 400)
    raise

When it happens

Trigger: Crawling a URL whose host is unreachable, an expired/self-signed TLS cert, a 10-second timeout on a slow server, or a server that drops the connection — any requests.RequestException during the pre-crawl redirect check.

Common situations: Intranet URLs not reachable from the RAGFlow container; firewalled hosts; sites with slow TTFB exceeding the hard-coded 10s timeout; transient network blips.

Related errors

• Execution timed out after {exec_timeout} seconds
• Function '{func.__name__}' timed out after {seconds} seconds
• Operation timed out after {seconds} seconds and {attempts} a
• Unexpected validation error: {e}
• WhatsApp session is not running.

AI-assisted analysis of infiniflow/ragflow@554fb1133a (2026-08-15). Data as JSON: /api/errors/8a43b5c34a419243. Report an issue: GitHub.