unclecode/crawl4ai · warning · ValueError

Unsupported export format: {format}

Error message

Unsupported export format: {format}

What it means

EgressBlocked ('URL blocked', raised from _resolve) when the hostname cannot be resolved at all — socket.getaddrinfo raises gaierror and the broker converts it to an opaque block. DNS failure is intentionally indistinguishable from a policy rejection so the API never explains why a host was refused (anti-SSRF information hygiene).

Source

Thrown at crawl4ai/adaptive_crawler.py:1806

        if not self.state or not self.state.knowledge_base:
            print("No knowledge base to export.")
            return
            
        filepath = Path(filepath)
        filepath.parent.mkdir(parents=True, exist_ok=True)
        
        if format == "jsonl":
            # Export as JSONL - one CrawlResult per line
            with open(filepath, 'w', encoding='utf-8') as f:
                for result in self.state.knowledge_base:
                    # Convert CrawlResult to dict
                    result_dict = self._crawl_result_to_export_dict(result)
                    # Write as single line JSON
                    f.write(json.dumps(result_dict, ensure_ascii=False) + '\n')
            
            print(f"Exported {len(self.state.knowledge_base)} documents to {filepath}")
        else:
            raise ValueError(f"Unsupported export format: {format}")
    
    def _crawl_result_to_export_dict(self, result) -> Dict[str, Any]:
        """Convert CrawlResult to a dictionary for export"""
        # Extract all available fields
        export_dict = {
            'url': getattr(result, 'url', ''),
            'timestamp': getattr(result, 'timestamp', None),
            'success': getattr(result, 'success', True),
            'query': self.state.query if self.state else '',
        }
        
        # Extract content
        if hasattr(result, 'markdown') and result.markdown:
            if hasattr(result.markdown, 'raw_markdown'):
                export_dict['content'] = result.markdown.raw_markdown
            else:
                export_dict['content'] = str(result.markdown)
        else:

View on GitHub (pinned to 7e80152142)

Solutions

1. Verify the hostname resolves from inside the container: docker exec <c> python -c "import socket; print(socket.getaddrinfo('host', 443))"
2. Fix the DNS setup (container --dns, resolv.conf, network) if resolution fails for known-good public domains
3. Correct typo'd/unregistered domains in the crawl list
4. For genuinely internal targets, run a deployment with ALLOW_INTERNAL enabled instead of trying to bypass the broker

Example fix

# before
urls = ["https://exmaple.com/page"]  # typo -> gaierror -> 'URL blocked'

# after
urls = ["https://example.com/page"]  # validate DNS first, then crawl

import socket
from urllib.parse import urlparse
def resolvable_public_url(url: str) -> bool:
    host = urlparse(url).hostname
    if not host:
        return False
    try:
        socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)
        return True
    except socket.gaierror:
        return False

from egress_broker import EgressBlocked
try:
    target = resolve_and_pin(url)
except EgressBlocked:
    skip_or_flag_url(url)  # opaque by design; do not retry blindly

When it happens

Trigger: Crawling a URL whose host has no DNS record, a typo'd domain, a resolver outage inside the container, or an internal-only DNS name while the deployment blocks internal egress — all produce EgressBlocked from the resolve step before any connection is attempted.

Common situations: Typo in the target domain; container DNS misconfiguration (bad --dns, broken resolv.conf); a newly registered/not-yet-propagated domain; attempting an intranet hostname from the public-egress deployment.

Related errors

• File not found: {filepath}
• field '{key}' is not permitted on {type_name} from an untrus
• Unsupported import format: {format}
• Invalid parameter(s) for {klass.__name__}: {invalid}
• type '{type_name}' may not be constructed from an untrusted

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/5dbcdd667c7822db. Report an issue: GitHub.