unclecode/crawl4ai · error · ValueError

CrawlerRunConfig must be provided

Error message

CrawlerRunConfig must be provided

What it means

Raised by BestFirstCrawlingStrategy.arun when the config argument is None. Although the signature declares config: Optional[CrawlerRunConfig] = None, the method body immediately rejects None because best-first crawling needs run settings (depth, stream mode, link filtering) to operate. It is a usage error, not a runtime failure.

Source

Thrown at crawl4ai/deep_crawling/bff_strategy.py:406

        Yields CrawlResults as they become available.
        """
        async for result in self._arun_best_first(start_url, crawler, config):
            yield result

    async def arun(
        self,
        start_url: str,
        crawler: AsyncWebCrawler,
        config: Optional[CrawlerRunConfig] = None,
    ) -> "RunManyReturn":
        """
        Main entry point for best-first crawling.
        
        Returns either a list (batch mode) or an async generator (stream mode)
        of CrawlResults.
        """
        if config is None:
            raise ValueError("CrawlerRunConfig must be provided")
        if config.stream:
            return self._arun_stream(start_url, crawler, config)
        else:
            return await self._arun_batch(start_url, crawler, config)

    async def shutdown(self) -> None:
        """
        Signal cancellation and clean up resources.
        """
        self._cancel_event.set()
        self.stats.end_time = datetime.now()

    def export_state(self) -> Optional[Dict[str, Any]]:
        """
        Export current crawl state for external persistence.

        Note: This returns the last captured state. For real-time state,
        use the on_state_change callback.

View on GitHub (pinned to 7e80152142)

Solutions

1. Pass a CrawlerRunConfig instance as the third argument: await strategy.arun(start_url, crawler, config=CrawlerRunConfig(...))
2. If using deep crawling normally, set deep_crawl_strategy=BestFirstCrawlingStrategy(...) on CrawlerRunConfig and call crawler.arun(url, config=config) so the library forwards the config itself
3. Set stream=True/False on the config explicitly if you need batch vs generator behavior

Example fix

// before
strategy = BestFirstCrawlingStrategy(...)
results = await strategy.arun(start_url, crawler)  # ValueError

// after
from crawl4ai import CrawlerRunConfig
config = CrawlerRunConfig(depth=2, stream=False)
results = await strategy.arun(start_url, crawler, config)

from crawl4ai import CrawlerRunConfig
assert config is not None, "BestFirstCrawlingStrategy.arun requires a CrawlerRunConfig"
if not isinstance(config, CrawlerRunConfig):
    raise TypeError(f"expected CrawlerRunConfig, got {type(config).__name__}")
results = await strategy.arun(start_url, crawler, config)

from crawl4ai.async_configs import CrawlerRunConfig

def is_run_config(cfg) -> bool:
    return isinstance(cfg, CrawlerRunConfig)

try:
    results = await strategy.arun(start_url, crawler, config)
except ValueError as e:
    if "must be provided" in str(e):
        raise RuntimeError("deep-crawl config missing") from e
    raise

When it happens

Trigger: Calling strategy.arun(start_url, crawler) or await strategy.arun(start_url, crawler, config=None) without a third argument. This happens when users port code from older crawl4ai versions where the config parameter was optional or positioned differently.

Common situations: Upgrading from an older crawl4ai release where arun(start_url, crawler) worked; constructing the strategy manually instead of via CrawlerRunConfig.deep_crawl_strategy and forgetting to forward the run config; copying examples that omit config.

Related errors

• CrawlerRunConfig must be provided
• Priority queue empty
• Invalid source(s): {invalid}. Valid: {VALID_SOURCES}
• too many hooks (max 10)
• unknown hook action {action!r}; allowed: {sorted(HOOK_REGIST

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/bf059a5aaab78cb5. Report an issue: GitHub.