infiniflow/ragflow · warning · BrowserFetchBusy

Too many concurrent browser fetch requests

Error message

Too many concurrent browser fetch requests

What it means

BrowserFetchBusy (RuntimeError subclass) raised by browser_fetch_slot in api/utils/web_utils.py:47-58. RAGFlow caps concurrent browser-based fetches (web crawling with headless-browser rendering) with a process-wide threading.BoundedSemaphore sized by BROWSER_FETCH_CONCURRENCY. When more fetches than slots arrive, callers block up to RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT seconds (default 5) waiting for a slot; if none frees up in time the context manager raises BrowserFetchBusy instead of queueing forever.

Source

Thrown at api/utils/web_utils.py:58

OTP_LENGTH = 4
OTP_TTL_SECONDS = 5 * 60  # valid for 5 minutes
ATTEMPT_LIMIT = 5  # maximum attempts
ATTEMPT_LOCK_SECONDS = 30 * 60  # lock for 30 minutes
RESEND_COOLDOWN_SECONDS = 60  # cooldown for 1 minute
BROWSER_FETCH_CONCURRENCY = max(1, int(os.getenv("RAGFLOW_BROWSER_FETCH_CONCURRENCY", "2")))
BROWSER_FETCH_ACQUIRE_TIMEOUT = float(os.getenv("RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT", "5"))
BROWSER_FETCH_TIMEOUT = float(os.getenv("RAGFLOW_BROWSER_FETCH_TIMEOUT", "60"))
_BROWSER_FETCH_SEMAPHORE = threading.BoundedSemaphore(BROWSER_FETCH_CONCURRENCY)


class BrowserFetchBusy(RuntimeError):
    pass


@contextmanager
def browser_fetch_slot(timeout: float = BROWSER_FETCH_ACQUIRE_TIMEOUT):
    if not _BROWSER_FETCH_SEMAPHORE.acquire(timeout=timeout):
        raise BrowserFetchBusy("Too many concurrent browser fetch requests")
    try:
        yield
    finally:
        _BROWSER_FETCH_SEMAPHORE.release()


from api.utils.file_response import (  # noqa: F401
    CONTENT_TYPE_MAP,
    FORCE_ATTACHMENT_CONTENT_TYPES,
    FORCE_ATTACHMENT_EXTENSIONS,
    agent_attachment_preview_path,
    apply_download_file_response_headers,
    apply_preview_file_response_headers,
    resolve_attachment_content_type,
    sanitize_content_disposition_filename,
    should_force_attachment,
)

View on GitHub (pinned to 554fb1133a)

Solutions

1. Throttle client-side concurrency so simultaneous browser fetches stay at or below BROWSER_FETCH_CONCURRENCY.
2. Raise RAGFLOW_BROWSER_FETCH_CONCURRENCY (and memory to match) if the host can support more headless-browser instances.
3. Raise RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT so bursty workloads wait for a slot instead of failing.
4. Retry the request after a short delay — slots free as in-flight fetches finish (each capped by RAGFLOW_BROWSER_FETCH_TIMEOUT).

Example fix

# before
results = await asyncio.gather(*[fetch_url(u) for u in urls])  # unbounded
# after
sem = asyncio.Semaphore(4)  # <= BROWSER_FETCH_CONCURRENCY
async def guarded(u):
    async with sem:
        return await fetch_url(u)
results = await asyncio.gather(*[guarded(u) for u in urls])

concurrency = int(os.getenv("RAGFLOW_BROWSER_FETCH_CONCURRENCY", "8"))
if in_flight_browser_fetches >= concurrency:
    # shed or queue locally instead of hitting the server's semaphore timeout
    await local_queue.put(request)

from api.utils.web_utils import BrowserFetchBusy
for attempt in range(3):
    try:
        return await fetch_with_browser(url)
    except BrowserFetchBusy:
        await asyncio.sleep(2 ** attempt)
raise RuntimeError("browser fetch saturated after retries")

When it happens

Trigger: Issuing more simultaneous browser fetch requests than BROWSER_FETCH_CONCURRENCY (e.g. N parallel document-ingestion jobs with web URLs requiring browser rendering) such that semaphore acquisition exceeds RAGFLOW_BROWSER_FETCH_ACQUIRE_TIMEOUT (default 5s). Long-running fetches (up to RAGFLOW_BROWSER_FETCH_TIMEOUT, default 60s) holding slots make this likelier.

Common situations: Bulk URL ingestion through the API/agent web-fetch tool with high parallelism; a burst of chat requests that each trigger browser fetch; a few slow pages hogging all slots while new requests queue; defaults tuned for single-user deployments used under load.

Related errors

• Failed to create search
• API rate limit exceeded. ${limit} requests/hour for unauthen
• Tenki rate limited, please retry: {exc}
• UCloud Agent Sandbox rate limited, please retry: {exc}
• GitHub search returned no items.

AI-assisted analysis of infiniflow/ragflow@554fb1133a (2026-08-15). Data as JSON: /api/errors/164a07c513c4018e. Report an issue: GitHub.