unclecode/crawl4ai · error · ValueError

Invalid URL, make sure the URL is a non-empty string

Error message

Invalid URL, make sure the URL is a non-empty string

What it means

AsyncWebCrawler.arun(url) requires url to be a non-empty str. The check runs after auto-start, before cache context creation, and rejects non-str types (None, bytes, ParseResult) and empty strings with this ValueError. It guards the very first step of a crawl.

Source

Thrown at crawl4ai/async_webcrawler.py:252

            result = await crawler.arun(url="https://example.com", config=config)

        Args:
            url: The URL to crawl (http://, https://, file://, or raw:)
            config: Configuration object controlling crawl behavior
            [other parameters maintained for backwards compatibility]

        Returns:
            CrawlResultContainer: A single-result container that proxies
                attribute access to the underlying CrawlResult for backwards
                compatibility (e.g. result.markdown, result.html).
        """
        # Auto-start if not ready
        if not self.ready:
            await self.start()

        config = config or CrawlerRunConfig()
        if not isinstance(url, str) or not url:
            raise ValueError(
                "Invalid URL, make sure the URL is a non-empty string")

        async with self._lock or self.nullcontext():
            try:
                self.logger.verbose = config.verbose

                # Default to ENABLED if no cache mode specified
                if config.cache_mode is None:
                    config.cache_mode = CacheMode.ENABLED

                # Create cache context
                cache_context = CacheContext(url, config.cache_mode, False)

                # Initialize processing variables
                async_response: AsyncCrawlResponse = None
                cached_result: CrawlResult = None
                screenshot_data = None
                pdf_data = None

View on GitHub (pinned to 7e80152142)

Solutions

1. Ensure the value passed is a str: coerce with str(url) or extract .get('url') properly from your data source.
2. Skip empty/None URLs before calling arun: if not url or not isinstance(url, str): continue.
3. If you have a parsed URL object, convert it back with url.geturl() / str(url).

Example fix

# before
for row in rows:
    result = await crawler.arun(row.get('url'))  # may be None

# after
for row in rows:
    url = row.get('url')
    if not isinstance(url, str) or not url.strip():
        continue
    result = await crawler.arun(url)

def assert_url(url):
    if not isinstance(url, str) or not url.strip():
        raise ValueError(f'invalid url: {url!r}')
    return url.strip()

from typing import Any
def is_crawlable_url(url: Any) -> bool:
    return isinstance(url, str) and bool(url.strip())

try:
    result = await crawler.arun(url)
except ValueError as e:
    if 'Invalid URL' in str(e):
        continue  # skip bad row in a batch loop
    raise

When it happens

Trigger: Calling arun(None) because a URL variable was never assigned; arun('') from an empty CSV/config cell; passing a yarl/urllib.parse.ParseResult or bytes object instead of a plain string; iterating a file whose lines are stripped to empty.

Common situations: Feeding crawler from data files where some rows lack URLs; loops that pass list elements of the wrong type; passing result of urlparse() directly; trailing whitespace-only strings still pass but '' does not.

Related errors

• Invalid source '{s}'. Valid sources are: {', '.join(valid_so
• `domain_or_domains` must be a string or a list of strings.
• Container not found: ${config.container_selector}
• Timeout after {timeout}ms waiting for selector '{wait_for}'
• [NSTProxy] token and channel_id are required

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/7ac23a6eeaae2ea9. Report an issue: GitHub.