unclecode/crawl4ai · warning · ValueError

too many webhook headers

Error message

too many webhook headers

What it means

ValueError from sanitize_webhook_headers when the user-supplied webhook headers dict exceeds _MAX_WEBHOOK_HEADERS (20) entries. A hard cap to bound request size and header-forwarding work.

Source

Thrown at deploy/docker/webhook.py:68

# Webhook request-header policy: user-controlled outbound headers could inject
# hop-by-hop / smuggling headers or CRLF. Allow only well-formed names, reject
# control chars in values, and deny sensitive/hop-by-hop names.
_WEBHOOK_HEADER_NAME = re.compile(r"^[A-Za-z0-9-]{1,64}$")
_WEBHOOK_DENY_HEADERS = {
    "host", "content-length", "transfer-encoding", "connection",
    "content-type", "proxy-authorization", "authorization", "cookie",
    "expect", "upgrade", "te", "trailer",
}
_MAX_WEBHOOK_HEADERS = 20
_MAX_WEBHOOK_HEADER_VALUE = 2048


def sanitize_webhook_headers(headers: Optional[Dict[str, str]]) -> Dict[str, str]:
    """Validate user-supplied webhook headers; raise ValueError on any bad one."""
    if not headers:
        return {}
    if len(headers) > _MAX_WEBHOOK_HEADERS:
        raise ValueError("too many webhook headers")
    clean: Dict[str, str] = {}
    for name, value in headers.items():
        if not isinstance(name, str) or not _WEBHOOK_HEADER_NAME.match(name):
            raise ValueError(f"invalid webhook header name: {name!r}")
        if name.lower() in _WEBHOOK_DENY_HEADERS:
            raise ValueError(f"webhook header not allowed: {name}")
        sval = str(value)
        if len(sval) > _MAX_WEBHOOK_HEADER_VALUE or any(c in sval for c in "\r\n\x00"):
            raise ValueError(f"invalid value for webhook header {name}")
        clean[name] = sval
    return clean


class WebhookDeliveryService:
    """Handles webhook delivery with exponential backoff retry logic."""

    def __init__(self, config: Dict):
        """

View on GitHub (pinned to 7e80152142)

Solutions

1. Trim the headers to the <=20 the receiver actually needs (usually just authorization + content-type + an idempotency key)
2. If more metadata must travel, put it in the webhook payload body, not headers

Example fix

# before
"headers": dict(all_outbound_headers)  # 30+ entries

# after
"headers": {
    "authorization": all_outbound_headers["Authorization"],
    "x-request-id": all_outbound_headers["X-Request-ID"],
}

assert isinstance(headers, dict) and 0 < len(headers) <= 20, f"webhook headers must be 1..20 entries, got {len(headers)}"

def is_valid_webhook_headers(h) -> bool:
    return isinstance(h, dict) and len(h) <= 20

When it happens

Trigger: POST /crawl with a webhook.headers object containing more than 20 keys.

Common situations: Forwarding an entire outbound request's header set (cookies, tracing, auth, telemetry) verbatim as webhook headers; programmatically generated header dicts (one per locale/feature flag) exceeding the cap.

Related errors

• invalid value for webhook header {name}
• URL must have a valid hostname
• invalid webhook header name: {name!r}
• webhook header not allowed: {name}
• Hook '{hook_name}' must be a callable function, got {type(ho

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/e7da545f2535f922. Report an issue: GitHub.