{"record":{"id":"bf6d60479c187bd6","repo":"unclecode/crawl4ai","slug":"field-key-is-not-permitted-on-type-name-from","errorCode":null,"errorMessage":"field '{key}' is not permitted on {type_name} from an untrusted request","messagePattern":"field '(.+?)' is not permitted on (.+?) from an untrusted request","errorType":"validation","errorClass":"UntrustedConfigError","httpStatus":null,"severity":"warning","filePath":"crawl4ai/async_configs.py","lineNumber":284,"sourceCode":"        \"check_robots_txt\", \"user_agent\", \"user_agent_mode\",\n        \"user_agent_generator_config\", \"url_matcher\", \"match_mode\", \"max_retries\",\n    },\n}\n\n# Upper bounds applied to attacker-influenced quantities after filtering.\n_MAX_TIMEOUT_MS = 60_000\n_MAX_SCROLL_STEPS = 1000\n_MAX_VIEWPORT = 4000\n\n\ndef _filter_untrusted_fields(type_name: str, params: dict) -> dict:\n    \"\"\"Drop non-allowlisted fields and raise on forbidden (power) fields.\"\"\"\n    forbidden = UNTRUSTED_FORBIDDEN_FIELDS.get(type_name, set())\n    allowlist = UNTRUSTED_FIELD_ALLOWLIST.get(type_name)  # None => keep all non-forbidden\n    out = {}\n    for key, value in params.items():\n        if key in forbidden:\n            raise UntrustedConfigError(\n                f\"field '{key}' is not permitted on {type_name} from an untrusted request\"\n            )\n        if allowlist is not None and key not in allowlist:\n            continue  # silently drop unknown/unsafe fields (forward-compatible)\n        out[key] = value\n    return out\n\n\ndef _clamp_untrusted(type_name: str, params: dict) -> dict:\n    \"\"\"Clamp attacker-influenced quantities to safe upper bounds.\"\"\"\n    def _cap_timeout(v):\n        # 0 historically meant \"no timeout\"; treat as the cap, never unbounded.\n        if not isinstance(v, (int, float)) or v <= 0:\n            return _MAX_TIMEOUT_MS\n        return min(int(v), _MAX_TIMEOUT_MS)\n\n    if type_name == \"CrawlerRunConfig\":\n        for f in (\"page_timeout\", \"wait_for_timeout\"):","sourceCodeStart":266,"sourceCodeEnd":302,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_configs.py#L266-L302","documentation":"EgressBlocked ('URL blocked') from resolve_and_pin when the URL's scheme is anything other than http or https (lowercased). The broker only mediates plain HTTP(S) egress; ftp:, file:, javascript:, data:, chrome:, ws:, etc. are refused before any host analysis.","triggerScenarios":"Passing a URL like file:///etc/passwd, ftp://host/file, or data:text/html,... to resolve_and_pin — typically from unvalidated user input or a redirect Location with an exotic scheme.","commonSituations":"User-submitted URL fields accepting arbitrary schemes; scraped hrefs containing javascript: or mailto: being fed back as crawl targets; redirect chains landing on a non-HTTP scheme.","solutions":["Normalize/validate crawl URLs to http/https before submission","Filter non-http(s) hrefs when building a link graph from scraped pages","Reject non-HTTP Location headers at redirect time instead of recursing into resolve_and_pin"],"exampleFix":"# before\nurls = [href for href in scraped_hrefs]  # may include mailto:, javascript:\n\n# after\nurls = [h for h in scraped_hrefs if h.lower().startswith((\"http://\", \"https://\"))]","handlingStrategy":"type-guard","validationCode":null,"typeGuard":"def is_http_url(url) -> bool:\n    s = str(url).lower()\n    return s.startswith(\"http://\") or s.startswith(\"https://\")","tryCatchPattern":"if not is_http_url(url):\n    discard(url, reason=\"non-http scheme\")\nelse:\n    target = resolve_and_pin(url)","preventionTips":["Filter scraped hrefs to http/https before feeding them back as crawl targets","Validate Location headers on redirects for scheme before re-validating","Reject file:, data:, javascript: schemes at intake"],"tags":["ssrf-protection","egress","scheme-validation","url-blocked"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}