unclecode/crawl4ai · error · ValueError

CrawlerRunConfig must be provided

Error message

CrawlerRunConfig must be provided

What it means

Deep-crawl strategies (BFS/DFS/BestBFS in deep_crawling/base_strategy.py) call crawler.arun(start_url, crawler, config) with a CrawlerRunConfig; passing config=None is rejected up front with this ValueError because every strategy branch (stream or batch) needs link filtering, depths, and stream settings from that config.

Source

Thrown at crawl4ai/deep_crawling/base_strategy.py:100

    async def arun(
        self,
        start_url: str,
        crawler: AsyncWebCrawler,
        config: Optional[CrawlerRunConfig] = None,
    ) -> RunManyReturn:
        """
        Traverse the given URL using the specified crawler.
        
        Args:
            start_url (str): The URL from which to start crawling.
            crawler (AsyncWebCrawler): The crawler instance to use.
            crawler_run_config (Optional[CrawlerRunConfig]): Crawler configuration.
        
        Returns:
            Union[CrawlResultT, List[CrawlResultT], AsyncGenerator[CrawlResultT, None]]
        """
        if config is None:
            raise ValueError("CrawlerRunConfig must be provided")

        if config.stream:
            return self._arun_stream(start_url, crawler, config)
        else:
            return await self._arun_batch(start_url, crawler, config)

    def __call__(self, start_url: str, crawler: AsyncWebCrawler, config: CrawlerRunConfig):
        return self.arun(start_url, crawler, config)

    @abstractmethod
    async def shutdown(self) -> None:
        """
        Clean up resources used by the deep crawl strategy.
        """
        pass

    @abstractmethod
    async def can_process_url(self, url: str, depth: int) -> bool:

View on GitHub (pinned to 7e80152142)

Solutions

1. Always pass a CrawlerRunConfig: await strategy.arun(start_url, crawler, CrawlerRunConfig(...)).
2. Prefer the high-level path: AsyncWebCrawler with CrawlerRunConfig(deep_cache... deep_crawl_strategy=strategy) so the crawler supplies its own config.
3. In custom strategies, thread the received config through instead of dropping it.

Example fix

# before
strategy = BFSDeepCrawlStrategy(max_depth=2, url_scorer=...
results = await strategy.arun(start_url, crawler)  # config=None -> ValueError

# after
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
results = await strategy.arun(start_url, crawler, run_config)

if config is None:
    config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)  # supply a default before calling

from crawl4ai import CrawlerRunConfig
def has_run_config(cfg) -> bool:
    return isinstance(cfg, CrawlerRunConfig)

try:
    results = await strategy.arun(start_url, crawler, config)
except ValueError as e:
    if 'CrawlerRunConfig must be provided' in str(e):
        results = await strategy.arun(start_url, crawler, CrawlerRunConfig())
    else:
        raise

When it happens

Trigger: Invoking strategy.arun(url, crawler) with two args (config defaults to None); passing config=None explicitly; building a custom loop that calls strategy.__call__ without forwarding the run config; calling DeepCrawlDecorator helpers where the config variable is unset.

Common situations: Writing custom deep-crawl strategies that forget to forward config; refactors where CrawlerRunConfig creation moved; calling the strategy directly instead of through crawler.arun(deep_crawl_strategy=...).

Related errors

• [NSTProxy] Invalid protocol: {protocol}
• concurrency must be positive
• timeout must be positive
• max_links must be positive
• score_threshold must be between 0.0 and 1.0

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/34ea5d00a8d292e5. Report an issue: GitHub.