unclecode/crawl4ai · error · ValueError

too many headers (max )

Error message

too many headers (max {_MAX_HEADERS})

What it means

SetHeadersParams' validator rejects a headers dict longer than _MAX_HEADERS = 20. This caps per-hook header overrides so a declarative crawl config cannot flood the browser with headers. Raised during validation of a set_headers hook spec in build_declarative_hooks.

Solutions

1. Trim the headers to the 20 you actually need (auth, UA, accept, cookies usually suffice).
2. If more are required, split across multiple set_headers specs — but note hooks per point are capped at 10 total.
3. Prefer setting session-level defaults in crawler_config instead of many per-hook headers.

Example fix

# before
{"action": "set_headers", "params": {"headers": {**thirty_headers}}}

# after
{"action": "set_headers", "params": {"headers": {k: v for k, v in thirty_headers.items() if k in needed_set}}}

MAX_HEADERS = 20

def valid_set_headers(spec: dict) -> bool:
    if spec.get("action") != "set_headers":
        return False
    h = spec.get("params", {}).get("headers")
    return isinstance(h, dict) and 1 <= len(h) <= MAX_HEADERS

def is_bounded_header_dict(v) -> bool:
    return isinstance(v, dict) and len(v) <= 20

try:
    hooks = build_declarative_hooks(specs)
except HookValidationError as e:
    if "too many headers" in str(e):
        trim_headers_to_top_priority(specs)  # then rebuild
    else:
        raise

When it happens

Trigger: Submitting a set_headers hook with a dict of more than 20 header name/value pairs in one spec.

Common situations: Forwarding a large corporate proxy/auth header set; pasting a whole browser profile's headers into the hook config; splitting work across multiple specs without realizing the cap is per-spec.

Related errors

• invalid params for hook
• unsupported resource_types
• control characters in value for header
• Hook ' ' must be a callable function, got
• invalid header name

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/417c5d68076727f3. Report an issue: GitHub.

Appendix: source

Thrown at deploy/docker/hook_registry.py:73

    value: str = Field(..., max_length=4096)
    domain: str = Field(..., min_length=1, max_length=253)
    path: str = "/"
    secure: bool = True
    httpOnly: bool = False


class AddCookiesParams(BaseModel):
    cookies: List[_Cookie] = Field(..., min_length=1, max_length=_MAX_COOKIES)


class SetHeadersParams(BaseModel):
    headers: Dict[str, str]

    @field_validator("headers")
    @classmethod
    def _check(cls, v):
        if len(v) > _MAX_HEADERS:
            raise ValueError(f"too many headers (max {_MAX_HEADERS})")
        for name, value in v.items():
            if not _HEADER_NAME_RE.match(name):
                raise ValueError(f"invalid header name {name!r}")
            if any(c in value for c in "\r\n\x00"):
                raise ValueError(f"control characters in value for header {name!r}")
        return v


class ScrollToBottomParams(BaseModel):
    max_steps: int = Field(10, ge=1, le=_MAX_SCROLL_STEPS)
    delay_ms: int = Field(500, ge=0, le=_MAX_SCROLL_DELAY_MS)


class WaitForTimeoutParams(BaseModel):
    timeout_ms: int = Field(..., ge=0, le=_MAX_WAIT_MS)


# ───────────────────────── server-authored hook factories ─────────────────────────

View on GitHub (pinned to 7e80152142)