{"record":{"id":"8a43b5c34a419243","repo":"infiniflow/ragflow","slug":"failed-to-fetch-current-url-r-exc","errorCode":null,"errorMessage":"Failed to fetch {current_url!r}: {_exc}","messagePattern":"Failed to fetch (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"api/db/services/file_service.py","lineNumber":819,"sourceCode":"                # follows a server-sent redirect to an unvalidated (potentially\n                # internal) host. Each hop is SSRF-checked before being followed;\n                # the validated (hostname, ip) pairs are pinned via Chromium's\n                # --host-resolver-rules so the browser cannot re-resolve any of them\n                # through a fresh DNS query.\n                current_url = url\n                current_hostname, current_ip = FileService._validate_url_for_crawl(current_url)\n                # Accumulate MAP rules for every hostname we encounter in the chain.\n                host_pins: dict[str, str] = {current_hostname: current_ip}\n\n                for _ in range(_MAX_CRAWL_REDIRECTS):\n                    try:\n                        _resp = _requests.get(\n                            current_url,\n                            timeout=10,\n                            allow_redirects=False,\n                        )\n                    except _requests.RequestException as _exc:\n                        raise ValueError(f\"Failed to fetch {current_url!r}: {_exc}\") from _exc\n\n                    if _resp.status_code not in (301, 302, 303, 307, 308):\n                        break\n\n                    _location = _resp.headers.get(\"Location\")\n                    if not _location:\n                        break\n\n                    _next_url = _urljoin(current_url, _location)\n                    _next_hostname, _next_ip = FileService._validate_url_for_crawl(_next_url)\n                    host_pins[_next_hostname] = _next_ip\n                    current_url = _next_url\n                else:\n                    raise ValueError(f\"Exceeded {_MAX_CRAWL_REDIRECTS} redirects fetching {url!r}\")\n\n                # Build a single MAP rule string covering every validated hostname\n                # in the redirect chain. Chromium uses the pinned IP for each,\n                # skipping DNS entirely and eliminating the rebinding window.","sourceCodeStart":801,"sourceCodeEnd":837,"githubUrl":"https://github.com/infiniflow/ragflow/blob/554fb1133ac3861732235ad9c377eb5e0a770665/api/db/services/file_service.py#L801-L837","documentation":"Raised in the crawl path of FileService web-crawling when the manual redirect-following requests.get fails with a RequestException (DNS failure, connection refused, TLS error, timeout of 10s). The URL was already SSRF-validated by _validate_url_for_crawl; this error means the HTTP fetch itself failed.","triggerScenarios":"Crawling a URL whose host is unreachable, an expired/self-signed TLS cert, a 10-second timeout on a slow server, or a server that drops the connection — any requests.RequestException during the pre-crawl redirect check.","commonSituations":"Intranet URLs not reachable from the RAGFlow container; firewalled hosts; sites with slow TTFB exceeding the hard-coded 10s timeout; transient network blips.","solutions":["Verify the URL is reachable from the RAGFlow server container: curl -I <url>.","Fix DNS/network/firewall access between the server and the target host.","If the site is slow, the 10s timeout in this function must be raised in code (it is hard-coded).","Retry transient network failures; skip the URL if it is permanently unreachable."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import requests as r\ntry:\n    probe = r.head(url, timeout=10, allow_redirects=False)\nexcept r.RequestException:\n    return json_error_response('URL unreachable from server', 400)","typeGuard":"def is_reachable_url(url: str, timeout: float = 5.0) -> bool:\n    try:\n        r = _requests.get(url, timeout=timeout, allow_redirects=False)\n        return r.status_code < 500\n    except _requests.RequestException:\n        return False","tryCatchPattern":"try:\n    FileService.web_crawl(url)\nexcept ValueError as e:\n    if str(e).startswith('Failed to fetch'):\n        return json_error_response('target URL unreachable', 400)\n    raise","preventionTips":["Pre-check URL reachability from the RAGFlow container (network/DNS/TLS).","Remember the fetch timeout is hard-coded at 10s; slow sites will fail.","Add intranet hosts to container DNS/hosts or firewall allowlists before crawling them."],"tags":["network","crawl","timeout","requests","url-fetch"],"backgroundTag":null,"analyzedSha":"554fb1133ac3861732235ad9c377eb5e0a770665","analyzedAt":"2026-08-15T09:20:16.380Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}