unclecode/crawl4ai · error · HTTPException

Rejected request: {e}

Error message

Rejected request: {e}

What it means

HTTP 400 'Rejected request: {e}' from the main crawl handler (deploy/docker/api.py:813): the request body contained config that tripped server-side trust validation — UntrustedConfigError (client tried to set forbidden power-fields like cookies/storage_state/headers beyond policy, or construct disallowed types) or HookValidationError (declarative hook spec with unknown action or invalid params). The monitor records the rejection; the message names the offending item.

Source

Thrown at deploy/docker/api.py:813

            pass

        # Add hooks information if hooks were used
        if hooks_config:
            response["hooks"] = hooks_status

        return response

    except (UntrustedConfigError, HookValidationError) as e:
        # An untrusted request body tried to set a forbidden power-field,
        # construct a disallowed type, or specify an invalid hook. Client error.
        try:
            from monitor import get_monitor
            await get_monitor().track_request_end(
                request_id, success=False, error=str(e), status_code=400
            )
        except:
            pass
        raise HTTPException(status_code=400, detail=f"Rejected request: {e}")

    except asyncio.TimeoutError:
        # Per-crawl wall-clock deadline exceeded.
        raise HTTPException(status_code=504, detail="Crawl exceeded the time limit")

    except HTTPException:
        # Deliberate status (e.g. 400 SSRF "URL blocked") must pass through
        # rather than be genericized to 500 by the handler below.
        raise

    except Exception as e:
        logger.error(f"Crawl error: {str(e)}", exc_info=True)

        # Track request error
        try:
            from monitor import get_monitor
            await get_monitor().track_request_end(
                request_id, success=False, error=str(e), status_code=500

View on GitHub (pinned to 7e80152142)

Solutions

1. Read detail — it includes the specific exception text naming the forbidden field or invalid hook.
2. Strip power-fields from the request; set them server-side in the trusted config instead.
3. Check the server's declarative-hook schema for allowed actions/params and conform the hooks block.
4. Align client and server versions so accepted config fields match.

Example fix

# before
body = {"url": u, "crawler_config": {"cookies": cookies, "verbose": True}}

# after
body = {"url": u, "crawler_config": {"verbose": True}}  # cookies set via trusted server config
r = client.post("/crawl", json=body)
if r.status_code == 400:
    print(r.json()["detail"])  # names the forbidden field

FORBIDDEN_FIELDS = {"cookies", "storage_state", "proxy", "user_agent", "extra_args"}

def crawl_body_is_safe(body: dict) -> bool:
    cfg = body.get("crawler_config", {})
    if FORBIDDEN_FIELDS & set(cfg.get("browser", {})):
        return False
    if FORBIDDEN_FIELDS & set(cfg):
        return False
    return all(h.get("action") in ALLOWED_HOOK_ACTIONS for h in body.get("hooks", {}).get("hooks", []))

r = await client.post("/crawl", json=body)
if r.status_code == 400 and "Rejected request" in r.json().get("detail", ""):
    detail = r.json()["detail"]
    strip_forbidden(body, detail)   # parse named field from message, remove, resubmit once
    r = await client.post("/crawl", json=body)

When it happens

Trigger: POST /crawl with a crawler_config JSON that includes fields on the server's forbidden list (e.g. browser-level extra_args, proxy settings, storage_state); hooks spec using an action name the server does not whitelist; wrong param types in a hook action (string where int required).

Common situations: Porting a working local crawl4ai CrawlerRunConfig verbatim into the Docker API body; newer client sending config fields this (older) server build forbids; probing users testing what the sandbox permits.

Related errors

• Failed to evaluate wait condition: {str(e)}
• invalid header name {name!r}
• control characters in value for header {name!r}
• Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enabl
• Rejected config: {e}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/895faee2da52331c. Report an issue: GitHub.