{"record":{"id":"34902fa90f5dc6dc","repo":"infiniflow/ragflow","slug":"exceeded-max-crawl-redirects-redirects-fetching","errorCode":null,"errorMessage":"Exceeded {_MAX_CRAWL_REDIRECTS} redirects fetching {url!r}","messagePattern":"Exceeded (.+?) redirects fetching (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"api/db/services/file_service.py","lineNumber":833,"sourceCode":"                            timeout=10,\n                            allow_redirects=False,\n                        )\n                    except _requests.RequestException as _exc:\n                        raise ValueError(f\"Failed to fetch {current_url!r}: {_exc}\") from _exc\n\n                    if _resp.status_code not in (301, 302, 303, 307, 308):\n                        break\n\n                    _location = _resp.headers.get(\"Location\")\n                    if not _location:\n                        break\n\n                    _next_url = _urljoin(current_url, _location)\n                    _next_hostname, _next_ip = FileService._validate_url_for_crawl(_next_url)\n                    host_pins[_next_hostname] = _next_ip\n                    current_url = _next_url\n                else:\n                    raise ValueError(f\"Exceeded {_MAX_CRAWL_REDIRECTS} redirects fetching {url!r}\")\n\n                # Build a single MAP rule string covering every validated hostname\n                # in the redirect chain. Chromium uses the pinned IP for each,\n                # skipping DNS entirely and eliminating the rebinding window.\n                _map_rules = \",\".join(f\"MAP {h} {ip}\" for h, ip in host_pins.items())\n\n                from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CrawlResult, DefaultMarkdownGenerator, PruningContentFilter\n\n                filename = re.sub(r\"\\?.*\", \"\", url.split(\"/\")[-1])\n\n                async def adownload():\n                    browser_config = BrowserConfig(\n                        headless=True,\n                        verbose=False,\n                        extra_args=[f\"--host-resolver-rules={_map_rules}\"],\n                    )\n                    async with AsyncWebCrawler(config=browser_config) as crawler:\n                        crawler_config = CrawlerRunConfig(markdown_generator=DefaultMarkdownGenerator(content_filter=PruningContentFilter()), pdf=True, screenshot=False)","sourceCodeStart":815,"sourceCodeEnd":851,"githubUrl":"https://github.com/infiniflow/ragflow/blob/554fb1133ac3861732235ad9c377eb5e0a770665/api/db/services/file_service.py#L815-L851","documentation":"Raised in the crawl path when the redirect chain for a URL exceeds _MAX_CRAWL_REDIRECTS (10) hops. RAGFlow manually follows 301/302/303/307/308 redirects to pin each hostname-to-IP (SSRF protection via Chromium host-resolver rules); a chain longer than 10 is treated as malicious or broken and aborted.","triggerScenarios":"Crawling a URL behind a redirect chain longer than 10 (CDN hops, login redirects, geo-redirect loops counting toward the limit, or an intentional open-redirect chain).","commonSituations":"URL shorteners stacked on each other; sites redirecting http->https->www->auth->back in a loop; misconfigured servers bouncing between two URLs.","solutions":["Resolve the final URL manually (curl -sIL <url> | grep -i location) and crawl the destination directly.","Eliminate redundant redirect hops on the target server if you control it.","Detect and fix redirect loops in the site's configuration.","Only if legitimate chains longer than 10 are required, raise _MAX_CRAWL_REDIRECTS (file_service.py:797) — but long chains are often a redirect attack, so prefer the direct URL."],"exampleFix":"# before\ncrawl('https://short.link/a')  # 11-hop chain -> ValueError\n\n# after\n# follow redirects out-of-band, crawl final destination\ncrawl('https://example.com/docs/final-page')","handlingStrategy":"validation","validationCode":"MAX_HOPS = 10\nu, hops = url, 0\nwhile hops < MAX_HOPS:\n    resp = requests.head(u, timeout=10, allow_redirects=False)\n    if resp.status_code not in (301, 302, 303, 307, 308):\n        break\n    u = urljoin(u, resp.headers.get('Location', ''))\n    hops += 1\nelse:\n    return json_error_response('redirect chain too long', 400)\nurl = u  # crawl resolved destination","typeGuard":null,"tryCatchPattern":"try:\n    FileService.web_crawl(url)\nexcept ValueError as e:\n    if 'redirects' in str(e):\n        # resolve final URL out-of-band and crawl that\n        final = resolve_final_url(url)\n        FileService.web_crawl(final)","preventionTips":["Resolve shortened URLs to their destination before submitting to the crawler.","Fix redirect loops on servers you control.","Treat long redirect chains as suspicious (open-redirect abuse) rather than raising the limit blindly."],"tags":["crawl","redirect","ssrf-protection","url","limit"],"backgroundTag":null,"analyzedSha":"554fb1133ac3861732235ad9c377eb5e0a770665","analyzedAt":"2026-08-15T09:20:16.380Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}