{"record":{"id":"417c5d68076727f3","repo":"unclecode/crawl4ai","slug":"too-many-headers-max-max-headers","errorCode":null,"errorMessage":"too many headers (max {_MAX_HEADERS})","messagePattern":"too many headers \\(max (.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":400,"severity":"error","filePath":"deploy/docker/hook_registry.py","lineNumber":73,"sourceCode":"    value: str = Field(..., max_length=4096)\n    domain: str = Field(..., min_length=1, max_length=253)\n    path: str = \"/\"\n    secure: bool = True\n    httpOnly: bool = False\n\n\nclass AddCookiesParams(BaseModel):\n    cookies: List[_Cookie] = Field(..., min_length=1, max_length=_MAX_COOKIES)\n\n\nclass SetHeadersParams(BaseModel):\n    headers: Dict[str, str]\n\n    @field_validator(\"headers\")\n    @classmethod\n    def _check(cls, v):\n        if len(v) > _MAX_HEADERS:\n            raise ValueError(f\"too many headers (max {_MAX_HEADERS})\")\n        for name, value in v.items():\n            if not _HEADER_NAME_RE.match(name):\n                raise ValueError(f\"invalid header name {name!r}\")\n            if any(c in value for c in \"\\r\\n\\x00\"):\n                raise ValueError(f\"control characters in value for header {name!r}\")\n        return v\n\n\nclass ScrollToBottomParams(BaseModel):\n    max_steps: int = Field(10, ge=1, le=_MAX_SCROLL_STEPS)\n    delay_ms: int = Field(500, ge=0, le=_MAX_SCROLL_DELAY_MS)\n\n\nclass WaitForTimeoutParams(BaseModel):\n    timeout_ms: int = Field(..., ge=0, le=_MAX_WAIT_MS)\n\n\n# ───────────────────────── server-authored hook factories ─────────────────────────","sourceCodeStart":55,"sourceCodeEnd":91,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/hook_registry.py#L55-L91","documentation":"SetHeadersParams' validator rejects a headers dict longer than _MAX_HEADERS = 20. This caps per-hook header overrides so a declarative crawl config cannot flood the browser with headers. Raised during validation of a set_headers hook spec in build_declarative_hooks.","triggerScenarios":"Submitting a set_headers hook with a dict of more than 20 header name/value pairs in one spec.","commonSituations":"Forwarding a large corporate proxy/auth header set; pasting a whole browser profile's headers into the hook config; splitting work across multiple specs without realizing the cap is per-spec.","solutions":["Trim the headers to the 20 you actually need (auth, UA, accept, cookies usually suffice).","If more are required, split across multiple set_headers specs — but note hooks per point are capped at 10 total.","Prefer setting session-level defaults in crawler_config instead of many per-hook headers."],"exampleFix":"# before\n{\"action\": \"set_headers\", \"params\": {\"headers\": {**thirty_headers}}}\n\n# after\n{\"action\": \"set_headers\", \"params\": {\"headers\": {k: v for k, v in thirty_headers.items() if k in needed_set}}}","handlingStrategy":"validation","validationCode":"MAX_HEADERS = 20\n\ndef valid_set_headers(spec: dict) -> bool:\n    if spec.get(\"action\") != \"set_headers\":\n        return False\n    h = spec.get(\"params\", {}).get(\"headers\")\n    return isinstance(h, dict) and 1 <= len(h) <= MAX_HEADERS","typeGuard":"def is_bounded_header_dict(v) -> bool:\n    return isinstance(v, dict) and len(v) <= 20","tryCatchPattern":"try:\n    hooks = build_declarative_hooks(specs)\nexcept HookValidationError as e:\n    if \"too many headers\" in str(e):\n        trim_headers_to_top_priority(specs)  # then rebuild\n    else:\n        raise","preventionTips":["Track header count in config templates; alert above ~15.","Consolidate related headers into one spec instead of many small ones.","Prefer crawler_config-level defaults for static headers."],"tags":["validation","pydantic","headers","hooks"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}