unclecode/crawl4ai · error · HTTPException

At least one URL required

Error message

At least one URL required

What it means

A 400 from POST /crawl: the CrawlRequestWithHooks body contained an empty or missing urls list. The endpoint requires at least one URL before it will load config, apply hooks, or crawl. It is raised before the hooks check and config provenance validation.

Source

Thrown at deploy/docker/server.py:880

async def metrics():
    return RedirectResponse(config["observability"]["prometheus"]["endpoint"])


@app.post("/crawl")
@limiter.limit(config["rate_limiting"]["default_limit"])
@mcp_tool("crawl")
async def crawl(
    request: Request,
    crawl_request: CrawlRequestWithHooks,
    _td: Dict = Depends(token_dep),
):
    """
    Crawl a list of URLs and return the results as JSON.
    For streaming responses, use /crawl/stream endpoint.
    Supports optional user-provided hook functions for customization.
    """
    if not crawl_request.urls:
        raise HTTPException(400, "At least one URL required")
    if crawl_request.hooks and not HOOKS_ENABLED:
        raise HTTPException(403, "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.")
    # Check whether it is a redirection for a streaming request
    try:
        crawler_config = CrawlerRunConfig.load(
            crawl_request.crawler_config, provenance=Provenance.UNTRUSTED
        )
    except UntrustedConfigError as e:
        raise HTTPException(400, f"Rejected config: {e}")
    if crawler_config.stream:
        return await stream_process(crawl_request=crawl_request)
    
    # Prepare hooks config if provided
    hooks_config = None
    if crawl_request.hooks:
        hooks_config = {
            'hooks': crawl_request.hooks.hooks,
            'timeout': crawl_request.hooks.timeout

View on GitHub (pinned to 7e80152142)

Solutions

1. Ensure the request body includes at least one URL: {'urls': ['https://example.com'], ...}.
2. Client-side, skip the call when the URL list is empty after filtering.
3. For streaming, the same requirement applies before redirect to /crawl/stream — populate urls first.

Example fix

# before
resp = requests.post(f'{base}/crawl', json={'urls': urls})  # urls may be []
# after
if not urls:
    raise ValueError('no URLs to crawl')
resp = requests.post(f'{base}/crawl', json={'urls': urls})

def crawl_body(urls: list[str], **kw) -> dict:
    urls = [u for u in urls if u and u.strip()]
    if not urls:
        raise ValueError('crawl requires at least one non-empty URL')
    return {'urls': urls, **kw}

def has_urls(urls: list[str] | None) -> bool:
    return isinstance(urls, list) and len([u for u in urls if u and u.strip()]) > 0

When it happens

Trigger: POST /crawl with {'urls': []}, {'urls': null}, or omitting urls entirely (pydantic default may be empty). Any subsequent fields (crawler_config, hooks) are irrelevant — urls is checked first.

Common situations: Batch pipelines where a URL-extraction step produced zero results but the job still fires; client defaulting to an empty list; filtering code that removes all URLs (e.g. dedupe or domain filter).

Related errors

• Container not found: ${config.container_selector}
• str(e)
• error_msg
• str(e)
• Invalid status: {status}. Must be one of: all, active, compl

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/dde18d2cfe299bf3. Report an issue: GitHub.