What causes "Crawl exceeded the time limit" in unclecode/crawl4ai?

Raise the per-crawl time limit (server config) if the target legitimately needs longer.

unclecode/crawl4ai · warning · HTTPException

Crawl exceeded the time limit

Error message

Crawl exceeded the time limit

What it means

HTTP 504 'Crawl exceeded the time limit' (deploy/docker/api.py:817): the per-crawl wall-clock deadline (asyncio timeout wrapping the crawl) expired before crawler.arun returned. The server enforces a maximum crawl duration; slow pages, deep waits (wait_for, delay_on_redirect), huge sitemaps, or a hung browser trip it. It is a gateway-style timeout: the crawl was killed, no result will appear for this request.

Source

Thrown at deploy/docker/api.py:817

            response["hooks"] = hooks_status

        return response

    except (UntrustedConfigError, HookValidationError) as e:
        # An untrusted request body tried to set a forbidden power-field,
        # construct a disallowed type, or specify an invalid hook. Client error.
        try:
            from monitor import get_monitor
            await get_monitor().track_request_end(
                request_id, success=False, error=str(e), status_code=400
            )
        except:
            pass
        raise HTTPException(status_code=400, detail=f"Rejected request: {e}")

    except asyncio.TimeoutError:
        # Per-crawl wall-clock deadline exceeded.
        raise HTTPException(status_code=504, detail="Crawl exceeded the time limit")

    except HTTPException:
        # Deliberate status (e.g. 400 SSRF "URL blocked") must pass through
        # rather than be genericized to 500 by the handler below.
        raise

    except Exception as e:
        logger.error(f"Crawl error: {str(e)}", exc_info=True)

        # Track request error
        try:
            from monitor import get_monitor
            await get_monitor().track_request_end(
                request_id, success=False, error=str(e), status_code=500
            )
        except:
            pass

View on GitHub (pinned to 7e80152142)

Solutions

Raise the per-crawl time limit (server config) if the target legitimately needs longer.
Tighten request-side knobs: shorter page_timeout, realistic wait_for with timeout, js_code that doesn't loop.
Retry once — genuinely hung sockets sometimes succeed on a fresh connection; use bounded backoff, not tight retry loops.
Pre-test the URL's load time externally and skip/queue known-slow targets instead of hitting the deadline.

Example fix

# before
r = client.post("/crawl", json={"url": slow_url})  # 504

# after
r = client.post("/crawl", json={
    "url": slow_url,
    "crawler_config": {"page_timeout": 60000, "wait_for": "body"},
})
if r.status_code == 504:
    time.sleep(5)
    r = client.post("/crawl", json={"url": slow_url, "crawler_config": {"page_timeout": 120000}})

Defensive patterns

Strategy: retry

Validate before calling

async def url_loads_within(url: str, budget_s: float) -> bool:
    try:
        await asyncio.wait_for(head_or_get(url), timeout=budget_s)
        return True
    except asyncio.TimeoutError:
        return False

# skip or lower wait_for requirements for URLs that fail the budget check

Try / catch

r = await client.post("/crawl", json=body)
if r.status_code == 504:
    body["crawler_config"] = {**body.get("crawler_config", {}), "page_timeout": 90000, "wait_for": "body"}
    await asyncio.sleep(3)
    r = await client.post("/crawl", json=body)  # single bounded retry, then give up

Prevention

Set page_timeout below the server's crawl deadline so browser timeouts fire first
Use broad wait_for targets (body) instead of specific selectors that may never appear
Pre-check heavy URLs and raise the server time limit or exclude them
Retry 504 at most once — a page that hangs usually hangs again

When it happens

Trigger: Crawling a page whose resources hang (server keeps connection open); configured wait_for='selector that never appears'; page_action/delay settings plus slow network exceeding the server's crawl timeout; browser deadlock in the container.

Common situations: Default timeout too small for legitimately heavy pages; crawling rate-limited hosts that drip bytes; playwright event never firing so navigation waits forever.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/1f03697db791e7d0. Report an issue: GitHub.