unclecode/crawl4ai · error · ValueError

CrawlerRunConfig must be provided

Error message

CrawlerRunConfig must be provided

What it means

Deep-crawl strategies (the traversal base class used by deep crawls) require a CrawlerRunConfig for every arun() call — it decides stream vs batch mode and carries per-crawl options. arun() accepts config as an Optional parameter but immediately raises ValueError when it is None.

Source

Thrown at crawl4ai/deep_crawling/base_strategy.py:100:9356

    async def arun(
        self,
        start_url: str,
        crawler: AsyncWebCrawler,
        config: Optional[CrawlerRunConfig] = None,
    ) -> RunManyReturn:
        """
        Traverse the given URL using the specified crawler.
        
        Args:
            start_url (str): The URL from which to start crawling.
            crawler (AsyncWebCrawler): The crawler instance to use.
            crawler_run_config (Optional[CrawlerRunConfig]): Crawler configuration.
        
        Returns:
            Union[CrawlResultT, List[CrawlResultT], AsyncGenerator[CrawlResultT, None]]
        """
        if config is None:
            raise ValueError("CrawlerRunConfig must be provided")

        if config.stream:
            return self._arun_stream(start_url, crawler, config)
        else:
            return await self._arun_batch(start_url, crawler, config)

    def __call__(self, start_url: str, crawler: AsyncWebCrawler, config: CrawlerRunConfig):
        return self.arun(start_url, crawler, config)

    @abstractmethod
    async def shutdown(self) -> None:
        """
        Clean up resources used by the deep crawl strategy.
        """
        pass

    @abstractmethod
    async def can_process_url(self, url: str, depth: int) -> bool:

View on GitHub (pinned to 7e80152142)

Solutions

1. Construct a CrawlerRunConfig (at minimum CrawlerRunConfig()) and pass it as the third argument: await strategy.arun(start_url, crawler, run_config)
2. If your wrapper accepts an optional config, default it to CrawlerRunConfig() instead of None
3. Set config.stream=True only if you will consume the async generator it returns

Example fix

# before
results = await strategy.arun(start_url, crawler)
# after
from crawl4ai import CrawlerRunConfig
results = await strategy.arun(start_url, crawler, CrawlerRunConfig(cache_mode="bypass", stream=False))

from crawl4ai import CrawlerRunConfig

def require_run_config(config):
    if config is None:
        return CrawlerRunConfig()
    return config

results = await strategy.arun(start_url, crawler, require_run_config(config))

from crawl4ai import CrawlerRunConfig

def is_run_config(obj) -> bool:
    return isinstance(obj, CrawlerRunConfig) and obj is not None

try:
    results = await strategy.arun(start_url, crawler, config)
except ValueError as e:
    if "CrawlerRunConfig" in str(e):
        results = await strategy.arun(start_url, crawler, CrawlerRunConfig())
    else:
        raise

When it happens

Trigger: Calling strategy.arun(start_url, crawler) with no third argument, or explicitly passing config=None; building the strategy's own CrawlerRunConfig inside a subclass but forgetting to forward it to super().arun().

Common situations: New users assuming the crawler instance alone carries the run config; wrapper code with an optional config parameter that defaults to None and is passed through unchecked; upgrades where a previously defaulted config became mandatory.

Related errors

• Container not found: ${config.container_selector}
• Hook '{hook_name}' must be a callable function, got {type(ho
• URL must have a valid hostname
• too many webhook headers
• invalid webhook header name: {name!r}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/0b82ec13d178597c. Report an issue: GitHub.