unclecode/crawl4ai · warning · HTTPException

Too many concurrent jobs for this caller

Error message

Too many concurrent jobs for this caller

What it means

HTTP 429 raised by the Docker crawl4ai server's queue dispatcher (deploy/docker/api.py:48) when work_queue.QuotaExceeded is thrown — the submitting caller (principal) already holds the maximum number of concurrent in-flight jobs allowed by its per-caller quota. It is a rate/quota signal, not a server failure: the request is rejected so the caller should slow down or await completion of running jobs.

Source

Thrown at deploy/docker/api.py:48

from llm_broker import LLMProviderNotAllowed
from crawl4ai.utils import perform_completion_with_backoff


def _enqueue_job(background_tasks, factory, principal=None):
    """Submit a background job to the bounded work queue (per-principal quota).

    Falls back to FastAPI BackgroundTasks when the queue isn't running (tests /
    no lifespan). Maps queue/quota limits to HTTP 503 / 429.
    """
    from work_queue import get_job_queue, QueueFull, QuotaExceeded
    q = get_job_queue()
    if q is None or not q.started:
        background_tasks.add_task(factory)
        return
    try:
        q.submit(factory, principal)
    except QuotaExceeded:
        raise HTTPException(status_code=429, detail="Too many concurrent jobs for this caller")
    except QueueFull:
        raise HTTPException(
            status_code=503, detail="Server busy, retry later",
            headers={"Retry-After": "5"},
        )


def _attach_declarative_hooks(crawler, hooks_config: dict) -> dict:
    """Build and attach server-authored hooks from declarative specs.

    Raises HookValidationError on an unknown action / invalid params, which the
    handlers map to HTTP 400.
    """
    specs = hooks_config.get("hooks", []) or []
    hooks = build_declarative_hooks(specs)
    for hook_point, fn in hooks.items():
        crawler.crawler_strategy.set_hook(hook_point, fn)
    return {"status": "success", "attached": list(hooks.keys())}

View on GitHub (pinned to 7e80152142)

Solutions

1. Client-side: cap concurrency at or below the quota (semaphore / smaller thread pool) — most common fix.
2. Honor 429 with exponential backoff and honor any Retry-After guidance, resubmitting when a slot frees.
3. If legitimate, raise the per-caller quota in the server's work_queue configuration.
4. Use distinct principals if multiple independent clients share one identity.

Example fix

# before
import asyncio
await asyncio.gather(*[client.post("/md", json=p) for p in payloads])  # 429s

# after
sem = asyncio.Semaphore(2)  # match server per-caller quota
async def one(p):
    async with sem:
        for attempt in range(5):
            r = await client.post("/md", json=p)
            if r.status_code != 429:
                return r
            await asyncio.sleep(2 ** attempt)

async def quota_slots_free(client, principal_concurrency: int, quota: int) -> bool:
    return principal_concurrency < quota  # track in-flight requests per token locally

resp = await client.post("/md", json=body)
if resp.status_code == 429:
    await asyncio.sleep(float(resp.headers.get("Retry-After", 1)) or 1)
    resp = await client.post("/md", json=body)  # bounded retry loop

When it happens

Trigger: A single API client firing more concurrent /md, /llm, or crawl requests than its configured quota (e.g. quota=2 and the client sends a 3rd before any finishes); a retry storm from a misconfigured client effectively multiplying concurrency; shared service account used by several workers whose combined load exceeds one principal's quota.

Common situations: Load-testing the server with one token; batch scripts with ThreadPoolExecutor sized above the quota; multiple developers/pods sharing the same credential (same principal sub) and hitting the aggregate cap.

Related errors

• Crawl request failed: {results['results'][0]['error_message'
• Invalid CSS selector: '{css_selector}'
• [NSTProxy] API Error: {data.get('msg', 'Unknown error')}
• Invalid CSS selector , No elements found for CSS selector: {
• Invalid CSS selector: {css_selector}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/4eb23fb6c302c883. Report an issue: GitHub.