unclecode/crawl4ai · error · ValueError

too many headers (max {_MAX_HEADERS})

Error message

too many headers (max {_MAX_HEADERS})

What it means

SetHeadersParams' validator rejects a headers dict longer than _MAX_HEADERS = 20. This caps per-hook header overrides so a declarative crawl config cannot flood the browser with headers. Raised during validation of a set_headers hook spec in build_declarative_hooks.

Source

Thrown at deploy/docker/hook_registry.py:73

    value: str = Field(..., max_length=4096)
    domain: str = Field(..., min_length=1, max_length=253)
    path: str = "/"
    secure: bool = True
    httpOnly: bool = False


class AddCookiesParams(BaseModel):
    cookies: List[_Cookie] = Field(..., min_length=1, max_length=_MAX_COOKIES)


class SetHeadersParams(BaseModel):
    headers: Dict[str, str]

    @field_validator("headers")
    @classmethod
    def _check(cls, v):
        if len(v) > _MAX_HEADERS:
            raise ValueError(f"too many headers (max {_MAX_HEADERS})")
        for name, value in v.items():
            if not _HEADER_NAME_RE.match(name):
                raise ValueError(f"invalid header name {name!r}")
            if any(c in value for c in "\r\n\x00"):
                raise ValueError(f"control characters in value for header {name!r}")
        return v


class ScrollToBottomParams(BaseModel):
    max_steps: int = Field(10, ge=1, le=_MAX_SCROLL_STEPS)
    delay_ms: int = Field(500, ge=0, le=_MAX_SCROLL_DELAY_MS)


class WaitForTimeoutParams(BaseModel):
    timeout_ms: int = Field(..., ge=0, le=_MAX_WAIT_MS)


# ───────────────────────── server-authored hook factories ─────────────────────────

View on GitHub (pinned to 7e80152142)

Solutions

1. Trim the headers to the 20 you actually need (auth, UA, accept, cookies usually suffice).
2. If more are required, split across multiple set_headers specs — but note hooks per point are capped at 10 total.
3. Prefer setting session-level defaults in crawler_config instead of many per-hook headers.

Example fix

# before
{"action": "set_headers", "params": {"headers": {**thirty_headers}}}

# after
{"action": "set_headers", "params": {"headers": {k: v for k, v in thirty_headers.items() if k in needed_set}}}

MAX_HEADERS = 20

def valid_set_headers(spec: dict) -> bool:
    if spec.get("action") != "set_headers":
        return False
    h = spec.get("params", {}).get("headers")
    return isinstance(h, dict) and 1 <= len(h) <= MAX_HEADERS

def is_bounded_header_dict(v) -> bool:
    return isinstance(v, dict) and len(v) <= 20

try:
    hooks = build_declarative_hooks(specs)
except HookValidationError as e:
    if "too many headers" in str(e):
        trim_headers_to_top_priority(specs)  # then rebuild
    else:
        raise

When it happens

Trigger: Submitting a set_headers hook with a dict of more than 20 header name/value pairs in one spec.

Common situations: Forwarding a large corporate proxy/auth header set; pasting a whole browser profile's headers into the hook config; splitting work across multiple specs without realizing the cap is per-spec.

Related errors

• unsupported resource_types {bad}; allowed: {sorted(_ALLOWED_
• invalid params for hook '{action}': {e}
• Hook '{hook_name}' must be a callable function, got {type(ho
• invalid header name {name!r}
• control characters in value for header {name!r}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/417c5d68076727f3. Report an issue: GitHub.