{"record":{"id":"5dbcdd667c7822db","repo":"unclecode/crawl4ai","slug":"unsupported-export-format-format","errorCode":null,"errorMessage":"Unsupported export format: {format}","messagePattern":"Unsupported export format: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"crawl4ai/adaptive_crawler.py","lineNumber":1806,"sourceCode":"        if not self.state or not self.state.knowledge_base:\n            print(\"No knowledge base to export.\")\n            return\n            \n        filepath = Path(filepath)\n        filepath.parent.mkdir(parents=True, exist_ok=True)\n        \n        if format == \"jsonl\":\n            # Export as JSONL - one CrawlResult per line\n            with open(filepath, 'w', encoding='utf-8') as f:\n                for result in self.state.knowledge_base:\n                    # Convert CrawlResult to dict\n                    result_dict = self._crawl_result_to_export_dict(result)\n                    # Write as single line JSON\n                    f.write(json.dumps(result_dict, ensure_ascii=False) + '\\n')\n            \n            print(f\"Exported {len(self.state.knowledge_base)} documents to {filepath}\")\n        else:\n            raise ValueError(f\"Unsupported export format: {format}\")\n    \n    def _crawl_result_to_export_dict(self, result) -> Dict[str, Any]:\n        \"\"\"Convert CrawlResult to a dictionary for export\"\"\"\n        # Extract all available fields\n        export_dict = {\n            'url': getattr(result, 'url', ''),\n            'timestamp': getattr(result, 'timestamp', None),\n            'success': getattr(result, 'success', True),\n            'query': self.state.query if self.state else '',\n        }\n        \n        # Extract content\n        if hasattr(result, 'markdown') and result.markdown:\n            if hasattr(result.markdown, 'raw_markdown'):\n                export_dict['content'] = result.markdown.raw_markdown\n            else:\n                export_dict['content'] = str(result.markdown)\n        else:","sourceCodeStart":1788,"sourceCodeEnd":1824,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/adaptive_crawler.py#L1788-L1824","documentation":"EgressBlocked ('URL blocked', raised from _resolve) when the hostname cannot be resolved at all — socket.getaddrinfo raises gaierror and the broker converts it to an opaque block. DNS failure is intentionally indistinguishable from a policy rejection so the API never explains why a host was refused (anti-SSRF information hygiene).","triggerScenarios":"Crawling a URL whose host has no DNS record, a typo'd domain, a resolver outage inside the container, or an internal-only DNS name while the deployment blocks internal egress — all produce EgressBlocked from the resolve step before any connection is attempted.","commonSituations":"Typo in the target domain; container DNS misconfiguration (bad --dns, broken resolv.conf); a newly registered/not-yet-propagated domain; attempting an intranet hostname from the public-egress deployment.","solutions":["Verify the hostname resolves from inside the container: docker exec <c> python -c \"import socket; print(socket.getaddrinfo('host', 443))\"","Fix the DNS setup (container --dns, resolv.conf, network) if resolution fails for known-good public domains","Correct typo'd/unregistered domains in the crawl list","For genuinely internal targets, run a deployment with ALLOW_INTERNAL enabled instead of trying to bypass the broker"],"exampleFix":"# before\nurls = [\"https://exmaple.com/page\"]  # typo -> gaierror -> 'URL blocked'\n\n# after\nurls = [\"https://example.com/page\"]  # validate DNS first, then crawl","handlingStrategy":"validation","validationCode":"import socket\nfrom urllib.parse import urlparse\ndef resolvable_public_url(url: str) -> bool:\n    host = urlparse(url).hostname\n    if not host:\n        return False\n    try:\n        socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)\n        return True\n    except socket.gaierror:\n        return False","typeGuard":null,"tryCatchPattern":"from egress_broker import EgressBlocked\ntry:\n    target = resolve_and_pin(url)\nexcept EgressBlocked:\n    skip_or_flag_url(url)  # opaque by design; do not retry blindly","preventionTips":["Pre-resolve hostnames from the same network namespace before submitting batches","Treat 'URL blocked' as opaque — distinguish DNS failure locally, not from the error","Keep container DNS (resolv.conf, --dns) correctly configured"],"tags":["ssrf-protection","egress","dns","url-blocked"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}