unclecode/crawl4ai · warning · HTTPException

Server busy, retry later

Error message

Server busy, retry later

What it means

HTTP 503 with Retry-After: 5 raised by the Docker server queue dispatcher (deploy/docker/api.py:50) when work_queue.QueueFull is thrown — the global job queue (not per-caller) is at capacity because the server is saturated. Unlike 147 this is server-wide backpressure: no config change on the client's request will help; the server simply has no queue slot right now.

Source

Thrown at deploy/docker/api.py:50

def _enqueue_job(background_tasks, factory, principal=None):
    """Submit a background job to the bounded work queue (per-principal quota).

    Falls back to FastAPI BackgroundTasks when the queue isn't running (tests /
    no lifespan). Maps queue/quota limits to HTTP 503 / 429.
    """
    from work_queue import get_job_queue, QueueFull, QuotaExceeded
    q = get_job_queue()
    if q is None or not q.started:
        background_tasks.add_task(factory)
        return
    try:
        q.submit(factory, principal)
    except QuotaExceeded:
        raise HTTPException(status_code=429, detail="Too many concurrent jobs for this caller")
    except QueueFull:
        raise HTTPException(
            status_code=503, detail="Server busy, retry later",
            headers={"Retry-After": "5"},
        )


def _attach_declarative_hooks(crawler, hooks_config: dict) -> dict:
    """Build and attach server-authored hooks from declarative specs.

    Raises HookValidationError on an unknown action / invalid params, which the
    handlers map to HTTP 400.
    """
    specs = hooks_config.get("hooks", []) or []
    hooks = build_declarative_hooks(specs)
    for hook_point, fn in hooks.items():
        crawler.crawler_strategy.set_hook(hook_point, fn)
    return {"status": "success", "attached": list(hooks.keys())}
from crawl4ai.content_filter_strategy import (
    PruningContentFilter,

View on GitHub (pinned to 7e80152142)

Solutions

1. Retry after the advertised delay (respect Retry-After: 5) with bounded exponential backoff and jitter.
2. Scale the deployment (more replicas/workers) or raise queue capacity in work_queue config if saturation is steady-state.
3. Reduce request cost: fewer pages per request, tighter crawler timeouts, LLM filter off where not needed.
4. Add a client-side circuit breaker so sustained 503s shed load instead of hammering.

Example fix

# before
r = client.post("/crawl", json=body)
assert r.status_code == 200

# after
for attempt in range(6):
    r = client.post("/crawl", json=body)
    if r.status_code != 503:
        break
    time.sleep(int(r.headers.get("Retry-After", 2 ** attempt)))

for attempt in range(5):
    resp = await client.post("/crawl", json=body)
    if resp.status_code != 503:
        break
    await asyncio.sleep(int(resp.headers.get("Retry-After", 2 ** attempt)) + random.random())

When it happens

Trigger: Aggregate concurrent crawl jobs across all callers reach the queue's max size; heavy LLM extraction jobs occupying workers while new /crawl requests arrive; undersized deployment (few workers, small queue) under burst traffic; downstream crawler/slowness (large pages, network latency) causing queue buildup.

Common situations: Public deployment absorbing a traffic spike; queue size left at default in a small container; crawls of slow domains stalling worker turnover.

Related errors

• Too many concurrent jobs for this caller
• result.error_message
• Crawl request failed: {results['results'][0]['error_message'

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/35714bd83701de38. Report an issue: GitHub.