unclecode/crawl4ai · warning · FileNotFoundError

File not found: {filepath}

Error message

File not found: {filepath}

What it means

EgressBlocked ('URL blocked') from assert_host_allowed when the host argument is empty/None. The broker treats a missing hostname as unconditionally blocked — there is nothing to resolve and the request must not proceed. It is a guard against callers passing malformed URLs downstream of parsing.

Source

Thrown at crawl4ai/adaptive_crawler.py:1854

        if self.state:
            export_dict['crawl_metadata'] = {
                'crawl_order': self.state.crawl_order.index(export_dict['url']) + 1 if export_dict['url'] in self.state.crawl_order else 0,
                'confidence_at_crawl': self.state.metrics.get('confidence', 0),
                'total_documents': self.state.total_documents
            }
        
        return export_dict
    
    async def import_knowledge_base(self, filepath: Union[str, Path], format: str = "jsonl") -> None:
        """Import a knowledge base from a file
        
        Args:
            filepath: Path to the file to import
            format: Import format - currently supports 'jsonl'
        """
        filepath = Path(filepath)
        if not filepath.exists():
            raise FileNotFoundError(f"File not found: {filepath}")
        
        if format == "jsonl":
            imported_results = []
            with open(filepath, 'r', encoding='utf-8') as f:
                for line in f:
                    if line.strip():
                        data = json.loads(line)
                        # Convert back to a mock CrawlResult
                        mock_result = self._import_dict_to_crawl_result(data)
                        imported_results.append(mock_result)
            
            # Initialize state if needed
            if not self.state:
                self.state = CrawlState()
            
            # Add imported results
            self.state.knowledge_base.extend(imported_results)
            

View on GitHub (pinned to 7e80152142)

Solutions

1. Validate/normalize URLs before calling the egress layer: require a scheme and a non-empty hostname
2. Filter empty or scheme-only entries out of batch crawl lists client-side
3. Log the offending raw input where you validate so the malformed entry is identifiable

Example fix

# before
assert_host_allowed(urlparse(raw).hostname or "", 443)

# after
from urllib.parse import urlparse
p = urlparse(raw)
if not p.hostname:
    raise ValueError(f"malformed URL, no host: {raw!r}")
assert_host_allowed(p.hostname, p.port or 443)

from urllib.parse import urlparse
def has_http_host(url) -> bool:
    try:
        return bool(urlparse(str(url)).hostname)
    except ValueError:
        return False

if not has_http_host(url):
    raise ValueError(f"URL missing host: {url!r}")
assert_host_allowed(urlparse(url).hostname, urlparse(url).port or 80)

When it happens

Trigger: Calling assert_host_allowed('', port) or with None — typically because a URL string like 'http://', ':///path', or a parse artifact yielded an empty host. Also hit when code extracts host from user input without validating it first.

Common situations: User-submitted URL lists containing malformed entries; URL construction bugs producing scheme-only strings; empty-string defaults flowing in from config.

Related errors

• Unsupported export format: {format}
• field '{key}' is not permitted on {type_name} from an untrus
• type '{type_name}' may not be constructed from an untrusted
• Unsupported import format: {format}
• Invalid parameter(s) for {klass.__name__}: {invalid}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/dff4eb125d54434d. Report an issue: GitHub.