unclecode/crawl4ai · error · ValueError

Invalid URL, make sure the URL is a non-empty string

Error message

Invalid URL, make sure the URL is a non-empty string

What it means

ValueError from AsyncWebCrawler's crawl entry (apriori path): url must be a non-empty str. Raised after auto-start, before any lock acquisition or cache handling, so it fails fast on bad input.

Source

Thrown at crawl4ai/async_webcrawler.py:252:1595

            )
            result = await crawler.arun(url="https://example.com", crawler_config=config)

        Args:
            url: The URL to crawl (http://, https://, file://, or raw:)
            crawler_config: Configuration object controlling crawl behavior
            [other parameters maintained for backwards compatibility]

        Returns:
            CrawlResult: The result of crawling and processing
        """
        # Auto-start if not ready
        if not self.ready:
            await self.start()

        config = config or CrawlerRunConfig()
        if not isinstance(url, str) or not url:
            raise ValueError(
                "Invalid URL, make sure the URL is a non-empty string")

        async with self._lock or self.nullcontext():
            try:
                self.logger.verbose = config.verbose

                # Default to ENABLED if no cache mode specified
                if config.cache_mode is None:
                    config.cache_mode = CacheMode.ENABLED

                # Create cache context
                cache_context = CacheContext(url, config.cache_mode, False)

                # Initialize processing variables
                async_response: AsyncCrawlResponse = None
                cached_result: CrawlResult = None
                screenshot_data = None
                pdf_data = None
                extracted_content = None

View on GitHub (pinned to 7e80152142)

Solutions

1. Filter inputs first: urls = [u for u in urls if isinstance(u, str) and u.strip()]
2. If a blank URL is expected data, log and skip it rather than aborting the whole batch
3. Double-check the arun/apriori call signature - a misplaced argument can land in url=

Example fix

# before
for u in rows:
    result = await crawler.arun(url=u, config=config)

# after
for u in rows:
    if not isinstance(u, str) or not u.strip():
        logger.warning("skipping empty url row")
        continue
    result = await crawler.arun(url=u, config=config)

assert isinstance(url, str) and url.strip(), f"url must be a non-empty string, got {url!r}"

def is_crawlable_url_value(u) -> bool:
    return isinstance(u, str) and bool(u.strip())

try:
    result = await crawler.arun(url=url, config=config)
except ValueError as e:
    if "non-empty string" in str(e):
        logger.warning("skipping bad url %r", url)
        result = None

When it happens

Trigger: await crawler.arun(url='') or url=None (or an int/list) - typically a URL built from a variable that ended up empty, or a loop iterating a list with an empty element.

Common situations: URLs scraped from a page where an anchor href was empty; CSV/database rows with blank URL columns; a chain that filters URLs and accidentally yields None; a positional argument shifted so a non-URL value lands in url.

Related errors

• Hook '{hook_name}' must be a callable function, got {type(ho
• unsupported resource_types {bad}; allowed: {sorted(_ALLOWED_
• URL must start with {schemes}
• URL must have a valid hostname
• too many webhook headers

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/d1f09a0dd446e4e6. Report an issue: GitHub.