{"record":{"id":"1f03697db791e7d0","repo":"unclecode/crawl4ai","slug":"crawl-exceeded-the-time-limit","errorCode":null,"errorMessage":"Crawl exceeded the time limit","messagePattern":"Crawl exceeded the time limit","errorType":"http","errorClass":"HTTPException","httpStatus":504,"severity":"warning","filePath":"deploy/docker/api.py","lineNumber":817,"sourceCode":"            response[\"hooks\"] = hooks_status\n\n        return response\n\n    except (UntrustedConfigError, HookValidationError) as e:\n        # An untrusted request body tried to set a forbidden power-field,\n        # construct a disallowed type, or specify an invalid hook. Client error.\n        try:\n            from monitor import get_monitor\n            await get_monitor().track_request_end(\n                request_id, success=False, error=str(e), status_code=400\n            )\n        except:\n            pass\n        raise HTTPException(status_code=400, detail=f\"Rejected request: {e}\")\n\n    except asyncio.TimeoutError:\n        # Per-crawl wall-clock deadline exceeded.\n        raise HTTPException(status_code=504, detail=\"Crawl exceeded the time limit\")\n\n    except HTTPException:\n        # Deliberate status (e.g. 400 SSRF \"URL blocked\") must pass through\n        # rather than be genericized to 500 by the handler below.\n        raise\n\n    except Exception as e:\n        logger.error(f\"Crawl error: {str(e)}\", exc_info=True)\n\n        # Track request error\n        try:\n            from monitor import get_monitor\n            await get_monitor().track_request_end(\n                request_id, success=False, error=str(e), status_code=500\n            )\n        except:\n            pass\n","sourceCodeStart":799,"sourceCodeEnd":835,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/api.py#L799-L835","documentation":"HTTP 504 'Crawl exceeded the time limit' (deploy/docker/api.py:817): the per-crawl wall-clock deadline (asyncio timeout wrapping the crawl) expired before crawler.arun returned. The server enforces a maximum crawl duration; slow pages, deep waits (wait_for, delay_on_redirect), huge sitemaps, or a hung browser trip it. It is a gateway-style timeout: the crawl was killed, no result will appear for this request.","triggerScenarios":"Crawling a page whose resources hang (server keeps connection open); configured wait_for='selector that never appears'; page_action/delay settings plus slow network exceeding the server's crawl timeout; browser deadlock in the container.","commonSituations":"Default timeout too small for legitimately heavy pages; crawling rate-limited hosts that drip bytes; playwright event never firing so navigation waits forever.","solutions":["Raise the per-crawl time limit (server config) if the target legitimately needs longer.","Tighten request-side knobs: shorter page_timeout, realistic wait_for with timeout, js_code that doesn't loop.","Retry once — genuinely hung sockets sometimes succeed on a fresh connection; use bounded backoff, not tight retry loops.","Pre-test the URL's load time externally and skip/queue known-slow targets instead of hitting the deadline."],"exampleFix":"# before\nr = client.post(\"/crawl\", json={\"url\": slow_url})  # 504\n\n# after\nr = client.post(\"/crawl\", json={\n    \"url\": slow_url,\n    \"crawler_config\": {\"page_timeout\": 60000, \"wait_for\": \"body\"},\n})\nif r.status_code == 504:\n    time.sleep(5)\n    r = client.post(\"/crawl\", json={\"url\": slow_url, \"crawler_config\": {\"page_timeout\": 120000}})","handlingStrategy":"retry","validationCode":"async def url_loads_within(url: str, budget_s: float) -> bool:\n    try:\n        await asyncio.wait_for(head_or_get(url), timeout=budget_s)\n        return True\n    except asyncio.TimeoutError:\n        return False\n\n# skip or lower wait_for requirements for URLs that fail the budget check","typeGuard":null,"tryCatchPattern":"r = await client.post(\"/crawl\", json=body)\nif r.status_code == 504:\n    body[\"crawler_config\"] = {**body.get(\"crawler_config\", {}), \"page_timeout\": 90000, \"wait_for\": \"body\"}\n    await asyncio.sleep(3)\n    r = await client.post(\"/crawl\", json=body)  # single bounded retry, then give up","preventionTips":["Set page_timeout below the server's crawl deadline so browser timeouts fire first","Use broad wait_for targets (body) instead of specific selectors that may never appear","Pre-check heavy URLs and raise the server time limit or exclude them","Retry 504 at most once — a page that hangs usually hangs again"],"tags":["http-504","timeout","crawl-limit","asyncio"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}