unclecode/crawl4ai · warning · ValueError

Unsupported import format: {format}

Error message

Unsupported import format: {format}

What it means

EgressBlocked ('URL blocked') from assert_host_allowed when the (lowercased) hostname is in the blocked-hostname set or starts with 'host.docker.internal'. These names resolve to container-host loopback/bridge addresses, which would let a crawled URL reach the host's own services — a classic SSRF vector — so they are rejected by name before DNS resolution.

Source

Thrown at crawl4ai/adaptive_crawler.py:1878

                    if line.strip():
                        data = json.loads(line)
                        # Convert back to a mock CrawlResult
                        mock_result = self._import_dict_to_crawl_result(data)
                        imported_results.append(mock_result)
            
            # Initialize state if needed
            if not self.state:
                self.state = CrawlState()
            
            # Add imported results
            self.state.knowledge_base.extend(imported_results)
            
            # Update state with imported data
            await self.strategy.update_state(self.state, imported_results)
            
            print(f"Imported {len(imported_results)} documents from {filepath}")
        else:
            raise ValueError(f"Unsupported import format: {format}")
    
    def _import_dict_to_crawl_result(self, data: Dict[str, Any]):
        """Convert imported dict back to a mock CrawlResult"""
        class MockMarkdown:
            def __init__(self, content):
                self.raw_markdown = content
        
        class MockCrawlResult:
            def __init__(self, data):
                self.url = data.get('url', '')
                self.markdown = MockMarkdown(data.get('content', ''))
                self.links = data.get('links', {})
                self.metadata = data.get('metadata', {})
                self.success = data.get('success', True)
                self.timestamp = data.get('timestamp')
        
        return MockCrawlResult(data)
    

View on GitHub (pinned to 7e80152142)

Solutions

1. Crawl the service via its publicly routable address instead of the docker-internal name
2. For local testing, run a deployment with ALLOW_INTERNAL=true (dev profile) rather than bypassing the broker
3. Remove container-internal hostnames from production crawl lists

Example fix

# before
urls = ["http://host.docker.internal:9000/health"]

# after
urls = ["https://my-service.example.com/health"]  # routable, non-internal target

def is_internal_name(host: str) -> bool:
    h = (host or "").lower()
    return not h or h.startswith("host.docker.internal") or h in BLOCKED_HOSTNAMES

from egress_broker import EgressBlocked
try:
    assert_host_allowed(host, port)
except EgressBlocked:
    route_to_internal_crawler(host)  # ALLOW_INTERNAL deployment

When it happens

Trigger: Crawling http://host.docker.internal:8080/... , localhost/127.0.0.1-style names in the blocked set, or any blocklisted hostname while ALLOW_INTERNAL is off. The name check fires even before getaddrinfo is called.

Common situations: Pointing the crawler at a service on the docker host during local testing; reusing docker-compose internal hostnames in the crawl list; a security posture change adding names to _BLOCKED_HOSTNAMES.

Related errors

• Invalid proxy string format: {proxy_str}
• Unsupported export format: {format}
• File not found: {filepath}
• Invalid parameter(s) for {klass.__name__}: {invalid}
• field '{key}' is not permitted on {type_name} from an untrus

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/64f359f4f88e9e81. Report an issue: GitHub.