unclecode/crawl4ai · error · HTTPException

Rejected config: {e}

Error message

Rejected config: {e}

What it means

A 400 from POST /crawl: CrawlerRunConfig.load() rejected the submitted crawler_config because it contained fields not allowed for Provenance.UNTRUSTED input, raising UntrustedConfigError. The endpoint deliberately loads client-supplied config in untrusted mode, blocking fields that could execute code or exfiltrate data server-side; the detail embeds the specific rejected field(s).

Source

Thrown at deploy/docker/server.py:889

    crawl_request: CrawlRequestWithHooks,
    _td: Dict = Depends(token_dep),
):
    """
    Crawl a list of URLs and return the results as JSON.
    For streaming responses, use /crawl/stream endpoint.
    Supports optional user-provided hook functions for customization.
    """
    if not crawl_request.urls:
        raise HTTPException(400, "At least one URL required")
    if crawl_request.hooks and not HOOKS_ENABLED:
        raise HTTPException(403, "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.")
    # Check whether it is a redirection for a streaming request
    try:
        crawler_config = CrawlerRunConfig.load(
            crawl_request.crawler_config, provenance=Provenance.UNTRUSTED
        )
    except UntrustedConfigError as e:
        raise HTTPException(400, f"Rejected config: {e}")
    if crawler_config.stream:
        return await stream_process(crawl_request=crawl_request)
    
    # Prepare hooks config if provided
    hooks_config = None
    if crawl_request.hooks:
        hooks_config = {
            'hooks': crawl_request.hooks.hooks,
            'timeout': crawl_request.hooks.timeout
        }
    
    results = await handle_crawl_request(
        urls=crawl_request.urls,
        browser_config=crawl_request.browser_config,
        crawler_config=crawl_request.crawler_config,
        config=config,
        hooks_config=hooks_config,
        crawler_configs=crawl_request.crawler_configs,

View on GitHub (pinned to 7e80152142)

Solutions

1. Parse the detail — the field names after 'Rejected config:' are exactly what to delete from crawler_config.
2. Strip privileged keys (js_code, hooks, and similar execution vectors) from client-submitted config before sending.
3. Use /execute_js (if enabled) for JS execution instead of smuggling js_code through /crawl's config.
4. Regenerate a minimal config from GET /schema and only override safe display/extraction options.

Example fix

# before
body = {'urls': [u], 'crawler_config': {'js_code': 'window.scrollTo(0,500)', 'word_count_threshold': 50}}
# after
body = {'urls': [u], 'crawler_config': {'word_count_threshold': 50}}

PRIVILEGED_KEYS = {'js_code', 'hooks', 'base_url'}  # extend from server's blocklist

def sanitize_crawler_config(cfg: dict) -> dict:
    bad = PRIVILEGED_KEYS & set(cfg or {})
    if bad:
        raise ValueError(f'remove privileged keys before /crawl: {sorted(bad)}')
    return cfg

def is_safe_crawler_config(cfg: dict) -> bool:
    return not ({'js_code', 'hooks', 'base_url'} & set(cfg or {}))

resp = requests.post(f'{BASE}/crawl', json=body, headers=hdrs)
if resp.status_code == 400 and resp.json().get('detail', '').startswith('Rejected config:'):
    detail = resp.json()['detail']
    # server names the offending keys; strip them and retry once
    for key in [w.strip(" '") for w in detail.split() if w in body.get('crawler_config', {})]:
        body['crawler_config'].pop(key, None)
    resp = requests.post(f'{BASE}/crawl', json=body, headers=hdrs)

When it happens

Trigger: POST /crawl with crawler_config containing privileged fields — e.g. js_code, hooks, base_url overrides, or any field blocklisted for untrusted provenance — in a server version that enforces provenance. The message after 'Rejected config:' names the offending key(s).

Common situations: Configs copied from older examples or local crawl4ai scripts that include js_code/hooks; version upgrade introducing provenance enforcement; automated config round-trips from /config/dump that carry now-restricted fields.

Related errors

• LLMConfig.api_token may not reference an environment variabl
• str(e)
• Container not found: ${config.container_selector}
• Invalid source(s): {invalid}. Valid: {VALID_SOURCES}
• str(e)

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/7393870a70275273. Report an issue: GitHub.