{"record":{"id":"35714bd83701de38","repo":"unclecode/crawl4ai","slug":"server-busy-retry-later","errorCode":null,"errorMessage":"Server busy, retry later","messagePattern":"Server busy, retry later","errorType":"http","errorClass":"HTTPException","httpStatus":503,"severity":"warning","filePath":"deploy/docker/api.py","lineNumber":50,"sourceCode":"\n\ndef _enqueue_job(background_tasks, factory, principal=None):\n    \"\"\"Submit a background job to the bounded work queue (per-principal quota).\n\n    Falls back to FastAPI BackgroundTasks when the queue isn't running (tests /\n    no lifespan). Maps queue/quota limits to HTTP 503 / 429.\n    \"\"\"\n    from work_queue import get_job_queue, QueueFull, QuotaExceeded\n    q = get_job_queue()\n    if q is None or not q.started:\n        background_tasks.add_task(factory)\n        return\n    try:\n        q.submit(factory, principal)\n    except QuotaExceeded:\n        raise HTTPException(status_code=429, detail=\"Too many concurrent jobs for this caller\")\n    except QueueFull:\n        raise HTTPException(\n            status_code=503, detail=\"Server busy, retry later\",\n            headers={\"Retry-After\": \"5\"},\n        )\n\n\ndef _attach_declarative_hooks(crawler, hooks_config: dict) -> dict:\n    \"\"\"Build and attach server-authored hooks from declarative specs.\n\n    Raises HookValidationError on an unknown action / invalid params, which the\n    handlers map to HTTP 400.\n    \"\"\"\n    specs = hooks_config.get(\"hooks\", []) or []\n    hooks = build_declarative_hooks(specs)\n    for hook_point, fn in hooks.items():\n        crawler.crawler_strategy.set_hook(hook_point, fn)\n    return {\"status\": \"success\", \"attached\": list(hooks.keys())}\nfrom crawl4ai.content_filter_strategy import (\n    PruningContentFilter,","sourceCodeStart":32,"sourceCodeEnd":68,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/api.py#L32-L68","documentation":"HTTP 503 with Retry-After: 5 raised by the Docker server queue dispatcher (deploy/docker/api.py:50) when work_queue.QueueFull is thrown — the global job queue (not per-caller) is at capacity because the server is saturated. Unlike 147 this is server-wide backpressure: no config change on the client's request will help; the server simply has no queue slot right now.","triggerScenarios":"Aggregate concurrent crawl jobs across all callers reach the queue's max size; heavy LLM extraction jobs occupying workers while new /crawl requests arrive; undersized deployment (few workers, small queue) under burst traffic; downstream crawler/slowness (large pages, network latency) causing queue buildup.","commonSituations":"Public deployment absorbing a traffic spike; queue size left at default in a small container; crawls of slow domains stalling worker turnover.","solutions":["Retry after the advertised delay (respect Retry-After: 5) with bounded exponential backoff and jitter.","Scale the deployment (more replicas/workers) or raise queue capacity in work_queue config if saturation is steady-state.","Reduce request cost: fewer pages per request, tighter crawler timeouts, LLM filter off where not needed.","Add a client-side circuit breaker so sustained 503s shed load instead of hammering."],"exampleFix":"# before\nr = client.post(\"/crawl\", json=body)\nassert r.status_code == 200\n\n# after\nfor attempt in range(6):\n    r = client.post(\"/crawl\", json=body)\n    if r.status_code != 503:\n        break\n    time.sleep(int(r.headers.get(\"Retry-After\", 2 ** attempt)))","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for attempt in range(5):\n    resp = await client.post(\"/crawl\", json=body)\n    if resp.status_code != 503:\n        break\n    await asyncio.sleep(int(resp.headers.get(\"Retry-After\", 2 ** attempt)) + random.random())","preventionTips":["Always honor the Retry-After header on 503","Add jitter to retries to avoid synchronized retry storms","Monitor server saturation and scale workers before steady-state 503s appear"],"tags":["http-503","backpressure","queue","retry-after","server"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}