unclecode/crawl4ai · warning · UntrustedConfigError

field '{key}' is not permitted on {type_name} from an untrus

Error message

field '{key}' is not permitted on {type_name} from an untrusted request

What it means

EgressBlocked ('URL blocked') from resolve_and_pin when the URL's scheme is anything other than http or https (lowercased). The broker only mediates plain HTTP(S) egress; ftp:, file:, javascript:, data:, chrome:, ws:, etc. are refused before any host analysis.

Source

Thrown at crawl4ai/async_configs.py:284

        "check_robots_txt", "user_agent", "user_agent_mode",
        "user_agent_generator_config", "url_matcher", "match_mode", "max_retries",
    },
}

# Upper bounds applied to attacker-influenced quantities after filtering.
_MAX_TIMEOUT_MS = 60_000
_MAX_SCROLL_STEPS = 1000
_MAX_VIEWPORT = 4000


def _filter_untrusted_fields(type_name: str, params: dict) -> dict:
    """Drop non-allowlisted fields and raise on forbidden (power) fields."""
    forbidden = UNTRUSTED_FORBIDDEN_FIELDS.get(type_name, set())
    allowlist = UNTRUSTED_FIELD_ALLOWLIST.get(type_name)  # None => keep all non-forbidden
    out = {}
    for key, value in params.items():
        if key in forbidden:
            raise UntrustedConfigError(
                f"field '{key}' is not permitted on {type_name} from an untrusted request"
            )
        if allowlist is not None and key not in allowlist:
            continue  # silently drop unknown/unsafe fields (forward-compatible)
        out[key] = value
    return out


def _clamp_untrusted(type_name: str, params: dict) -> dict:
    """Clamp attacker-influenced quantities to safe upper bounds."""
    def _cap_timeout(v):
        # 0 historically meant "no timeout"; treat as the cap, never unbounded.
        if not isinstance(v, (int, float)) or v <= 0:
            return _MAX_TIMEOUT_MS
        return min(int(v), _MAX_TIMEOUT_MS)

    if type_name == "CrawlerRunConfig":
        for f in ("page_timeout", "wait_for_timeout"):

View on GitHub (pinned to 7e80152142)

Solutions

1. Normalize/validate crawl URLs to http/https before submission
2. Filter non-http(s) hrefs when building a link graph from scraped pages
3. Reject non-HTTP Location headers at redirect time instead of recursing into resolve_and_pin

Example fix

# before
urls = [href for href in scraped_hrefs]  # may include mailto:, javascript:

# after
urls = [h for h in scraped_hrefs if h.lower().startswith(("http://", "https://"))]

def is_http_url(url) -> bool:
    s = str(url).lower()
    return s.startswith("http://") or s.startswith("https://")

if not is_http_url(url):
    discard(url, reason="non-http scheme")
else:
    target = resolve_and_pin(url)

When it happens

Trigger: Passing a URL like file:///etc/passwd, ftp://host/file, or data:text/html,... to resolve_and_pin — typically from unvalidated user input or a redirect Location with an exotic scheme.

Common situations: User-submitted URL fields accepting arbitrary schemes; scraped hrefs containing javascript: or mailto: being fed back as crawl targets; redirect chains landing on a non-HTTP scheme.

Related errors

• Unsupported export format: {format}
• File not found: {filepath}
• Unsupported import format: {format}
• Invalid parameter(s) for {klass.__name__}: {invalid}
• type '{type_name}' may not be constructed from an untrusted

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/bf6d60479c187bd6. Report an issue: GitHub.