infiniflow/ragflow · warning · ValueError

Exceeded {_MAX_CRAWL_REDIRECTS} redirects fetching {url!r}

Error message

Exceeded {_MAX_CRAWL_REDIRECTS} redirects fetching {url!r}

What it means

Raised in the crawl path when the redirect chain for a URL exceeds _MAX_CRAWL_REDIRECTS (10) hops. RAGFlow manually follows 301/302/303/307/308 redirects to pin each hostname-to-IP (SSRF protection via Chromium host-resolver rules); a chain longer than 10 is treated as malicious or broken and aborted.

Source

Thrown at api/db/services/file_service.py:833

                            timeout=10,
                            allow_redirects=False,
                        )
                    except _requests.RequestException as _exc:
                        raise ValueError(f"Failed to fetch {current_url!r}: {_exc}") from _exc

                    if _resp.status_code not in (301, 302, 303, 307, 308):
                        break

                    _location = _resp.headers.get("Location")
                    if not _location:
                        break

                    _next_url = _urljoin(current_url, _location)
                    _next_hostname, _next_ip = FileService._validate_url_for_crawl(_next_url)
                    host_pins[_next_hostname] = _next_ip
                    current_url = _next_url
                else:
                    raise ValueError(f"Exceeded {_MAX_CRAWL_REDIRECTS} redirects fetching {url!r}")

                # Build a single MAP rule string covering every validated hostname
                # in the redirect chain. Chromium uses the pinned IP for each,
                # skipping DNS entirely and eliminating the rebinding window.
                _map_rules = ",".join(f"MAP {h} {ip}" for h, ip in host_pins.items())

                from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CrawlResult, DefaultMarkdownGenerator, PruningContentFilter

                filename = re.sub(r"\?.*", "", url.split("/")[-1])

                async def adownload():
                    browser_config = BrowserConfig(
                        headless=True,
                        verbose=False,
                        extra_args=[f"--host-resolver-rules={_map_rules}"],
                    )
                    async with AsyncWebCrawler(config=browser_config) as crawler:
                        crawler_config = CrawlerRunConfig(markdown_generator=DefaultMarkdownGenerator(content_filter=PruningContentFilter()), pdf=True, screenshot=False)

View on GitHub (pinned to 554fb1133a)

Solutions

1. Resolve the final URL manually (curl -sIL <url> | grep -i location) and crawl the destination directly.
2. Eliminate redundant redirect hops on the target server if you control it.
3. Detect and fix redirect loops in the site's configuration.
4. Only if legitimate chains longer than 10 are required, raise _MAX_CRAWL_REDIRECTS (file_service.py:797) — but long chains are often a redirect attack, so prefer the direct URL.

Example fix

# before
crawl('https://short.link/a')  # 11-hop chain -> ValueError

# after
# follow redirects out-of-band, crawl final destination
crawl('https://example.com/docs/final-page')

MAX_HOPS = 10
u, hops = url, 0
while hops < MAX_HOPS:
    resp = requests.head(u, timeout=10, allow_redirects=False)
    if resp.status_code not in (301, 302, 303, 307, 308):
        break
    u = urljoin(u, resp.headers.get('Location', ''))
    hops += 1
else:
    return json_error_response('redirect chain too long', 400)
url = u  # crawl resolved destination

try:
    FileService.web_crawl(url)
except ValueError as e:
    if 'redirects' in str(e):
        # resolve final URL out-of-band and crawl that
        final = resolve_final_url(url)
        FileService.web_crawl(final)

When it happens

Trigger: Crawling a URL behind a redirect chain longer than 10 (CDN hops, login redirects, geo-redirect loops counting toward the limit, or an intentional open-redirect chain).

Common situations: URL shorteners stacked on each other; sites redirecting http->https->www->auth->back in a loop; misconfigured servers bouncing between two URLs.

Related errors

• Repository or path not found. Please check the URL and ensur
• No files found in the repository
• KEENABLE_API_URL must be an https:// URL with a host, got {b
• Failed to fetch {current_url!r}: {_exc}
• page_size must be less than or equal to {REST_API_MAX_PAGE_S

AI-assisted analysis of infiniflow/ragflow@554fb1133a (2026-08-15). Data as JSON: /api/errors/34902fa90f5dc6dc. Report an issue: GitHub.