unclecode/crawl4ai · warning · UntrustedConfigError

type '{type_name}' may not be constructed from an untrusted

Error message

type '{type_name}' may not be constructed from an untrusted request

What it means

EgressBlocked ('URL blocked') from resolve_and_pin when urlparse finds no hostname in the URL — e.g. 'http:///path', relative URLs, or bare strings. Like the empty-host check in assert_host_allowed, a URL without a host cannot be policy-checked or pinned, so it is rejected.

Source

Thrown at crawl4ai/async_configs.py:444

        isinstance(data, dict)
        and "type" in data
        and ("params" in data or (data["type"] == "dict" and "value" in data))
    ):
        # Handle plain dictionaries
        if data["type"] == "dict" and "value" in data:
            return {k: from_serializable_dict(v, provenance) for k, v in data["value"].items()}

        # Security: only allow known-safe types to be deserialized.
        # Unknown types (e.g. logging.Logger serialized by older clients) are
        # silently dropped (returned as None) instead of crashing the request.
        type_name = data["type"]
        if type_name not in ALLOWED_DESERIALIZE_TYPES:
            return None

        # Untrusted bodies may only construct the strict subset of types and
        # may not set forbidden power-fields.
        if provenance == Provenance.UNTRUSTED and type_name not in UNTRUSTED_ALLOWED_TYPES:
            raise UntrustedConfigError(
                f"type '{type_name}' may not be constructed from an untrusted request"
            )

        cls = None
        module_paths = ["crawl4ai"]
        for module_path in module_paths:
            try:
                mod = importlib.import_module(module_path)
                if hasattr(mod, type_name):
                    cls = getattr(mod, type_name)
                    break
            except (ImportError, AttributeError):
                continue

        if cls is not None:
            # Handle Enum
            if issubclass(cls, Enum):
                return cls(data["params"])

View on GitHub (pinned to 7e80152142)

Solutions

1. Fully resolve relative URLs against the page base with urllib.parse.urljoin before validation
2. Require parsed.hostname to be non-empty in your URL intake validation
3. Type-check inputs: only absolute http(s) URL strings should reach the crawl API

Example fix

# before
target = link  # '/docs/next' -> no host -> blocked

# after
from urllib.parse import urljoin
target = urljoin(page_url, link)  # 'https://site.com/docs/next'

from urllib.parse import urlparse
def is_absolute_http_url(u) -> bool:
    try:
        p = urlparse(str(u))
    except ValueError:
        return False
    return p.scheme in ("http", "https") and bool(p.hostname)

from urllib.parse import urljoin
u = urljoin(base_page_url, link)  # resolve relative first
if not is_absolute_http_url(u):
    discard(link)
else:
    target = resolve_and_pin(u)

When it happens

Trigger: Calling resolve_and_pin('http:///foo'), resolve_and_pin('/relative/path'), or a URL built by string concatenation that lost its authority component.

Common situations: Joining scraped relative links without urljoin; URLs assembled from config parts where the host segment is empty; passing an ID or slug where a URL was expected.

Related errors

• File not found: {filepath}
• Unsupported export format: {format}
• Unsupported import format: {format}
• Invalid parameter(s) for {klass.__name__}: {invalid}
• field '{key}' is not permitted on {type_name} from an untrus

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/8e4b8181ed045e27. Report an issue: GitHub.