{"record":{"id":"164a07c513c4018e","repo":"infiniflow/ragflow","slug":"too-many-concurrent-browser-fetch-requests","errorCode":null,"errorMessage":"Too many concurrent browser fetch requests","messagePattern":"Too many concurrent browser fetch requests","errorType":"exception","errorClass":"BrowserFetchBusy","httpStatus":null,"severity":"warning","filePath":"api/utils/web_utils.py","lineNumber":58,"sourceCode":"OTP_LENGTH = 4\nOTP_TTL_SECONDS = 5 * 60  # valid for 5 minutes\nATTEMPT_LIMIT = 5  # maximum attempts\nATTEMPT_LOCK_SECONDS = 30 * 60  # lock for 30 minutes\nRESEND_COOLDOWN_SECONDS = 60  # cooldown for 1 minute\nBROWSER_FETCH_CONCURRENCY = max(1, int(os.getenv(\"RAGFLOW_BROWSER_FETCH_CONCURRENCY\", \"2\")))\nBROWSER_FETCH_ACQUIRE_TIMEOUT = float(os.getenv(\"RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT\", \"5\"))\nBROWSER_FETCH_TIMEOUT = float(os.getenv(\"RAGFLOW_BROWSER_FETCH_TIMEOUT\", \"60\"))\n_BROWSER_FETCH_SEMAPHORE = threading.BoundedSemaphore(BROWSER_FETCH_CONCURRENCY)\n\n\nclass BrowserFetchBusy(RuntimeError):\n    pass\n\n\n@contextmanager\ndef browser_fetch_slot(timeout: float = BROWSER_FETCH_ACQUIRE_TIMEOUT):\n    if not _BROWSER_FETCH_SEMAPHORE.acquire(timeout=timeout):\n        raise BrowserFetchBusy(\"Too many concurrent browser fetch requests\")\n    try:\n        yield\n    finally:\n        _BROWSER_FETCH_SEMAPHORE.release()\n\n\nfrom api.utils.file_response import (  # noqa: F401\n    CONTENT_TYPE_MAP,\n    FORCE_ATTACHMENT_CONTENT_TYPES,\n    FORCE_ATTACHMENT_EXTENSIONS,\n    agent_attachment_preview_path,\n    apply_download_file_response_headers,\n    apply_preview_file_response_headers,\n    resolve_attachment_content_type,\n    sanitize_content_disposition_filename,\n    should_force_attachment,\n)\n","sourceCodeStart":40,"sourceCodeEnd":76,"githubUrl":"https://github.com/infiniflow/ragflow/blob/554fb1133ac3861732235ad9c377eb5e0a770665/api/utils/web_utils.py#L40-L76","documentation":"BrowserFetchBusy (RuntimeError subclass) raised by browser_fetch_slot in api/utils/web_utils.py:47-58. RAGFlow caps concurrent browser-based fetches (web crawling with headless-browser rendering) with a process-wide threading.BoundedSemaphore sized by BROWSER_FETCH_CONCURRENCY. When more fetches than slots arrive, callers block up to RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT seconds (default 5) waiting for a slot; if none frees up in time the context manager raises BrowserFetchBusy instead of queueing forever.","triggerScenarios":"Issuing more simultaneous browser fetch requests than BROWSER_FETCH_CONCURRENCY (e.g. N parallel document-ingestion jobs with web URLs requiring browser rendering) such that semaphore acquisition exceeds RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT (default 5s). Long-running fetches (up to RAGFLOW_BROWSER_FETCH_TIMEOUT, default 60s) holding slots make this likelier.","commonSituations":"Bulk URL ingestion through the API/agent web-fetch tool with high parallelism; a burst of chat requests that each trigger browser fetch; a few slow pages hogging all slots while new requests queue; defaults tuned for single-user deployments used under load.","solutions":["Throttle client-side concurrency so simultaneous browser fetches stay at or below BROWSER_FETCH_CONCURRENCY.","Raise RAGFLOW_BROWSER_FETCH_CONCURRENCY (and memory to match) if the host can support more headless-browser instances.","Raise RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT so bursty workloads wait for a slot instead of failing.","Retry the request after a short delay — slots free as in-flight fetches finish (each capped by RAGFLOW_BROWSER_FETCH_TIMEOUT)."],"exampleFix":"# before\nresults = await asyncio.gather(*[fetch_url(u) for u in urls])  # unbounded\n# after\nsem = asyncio.Semaphore(4)  # <= BROWSER_FETCH_CONCURRENCY\nasync def guarded(u):\n    async with sem:\n        return await fetch_url(u)\nresults = await asyncio.gather(*[guarded(u) for u in urls])","handlingStrategy":"retry","validationCode":"concurrency = int(os.getenv(\"RAGFLOW_BROWSER_FETCH_CONCURRENCY\", \"8\"))\nif in_flight_browser_fetches >= concurrency:\n    # shed or queue locally instead of hitting the server's semaphore timeout\n    await local_queue.put(request)","typeGuard":null,"tryCatchPattern":"from api.utils.web_utils import BrowserFetchBusy\nfor attempt in range(3):\n    try:\n        return await fetch_with_browser(url)\n    except BrowserFetchBusy:\n        await asyncio.sleep(2 ** attempt)\nraise RuntimeError(\"browser fetch saturated after retries\")","preventionTips":["Bound client concurrency to RAGFLOW_BROWSER_FETCH_CONCURRENCY.","Monitor in-flight browser fetches and shed load before the server's acquire timeout (default 5s) expires.","Tune RAGFLOW_BROWSER_FETCH_CONCURRENCY / RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT to workload reality."],"tags":["concurrency","rate-limit","web-fetch","semaphore","browser"],"backgroundTag":null,"analyzedSha":"554fb1133ac3861732235ad9c377eb5e0a770665","analyzedAt":"2026-08-15T09:20:16.380Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}