unclecode/crawl4ai · critical · ValueError

Unknown strategy: {strategy_name}

Error message

Unknown strategy: {strategy_name}

What it means

MemoryError raised by get_crawler before creating a new browser when container memory usage (get_container_memory_percent) is at or above MEM_LIMIT (config `crawler.memory_threshold_percent`, default 95%). It is a deliberate load-shed: the pool refuses to grow under memory pressure instead of letting the container get OOM-killed. Reusing a pooled crawler with the same signature bypasses the check; only new browser creation is gated.

Source

Thrown at crawl4ai/adaptive_crawler.py:1328

        self.state: Optional[CrawlState] = None
        
        # Track if we own the crawler (for cleanup)
        self._owns_crawler = crawler is None
    
    def _create_strategy(self, strategy_name: str) -> CrawlStrategy:
        """Create strategy instance based on name"""
        if strategy_name == "statistical":
            return StatisticalStrategy()
        elif strategy_name == "embedding":
            strategy = EmbeddingStrategy(
                embedding_model=self.config.embedding_model,
                llm_config=self.config.embedding_llm_config,
                query_llm_config=self.config.query_llm_config,
            )
            strategy.config = self.config  # Pass config to strategy
            return strategy
        else:
            raise ValueError(f"Unknown strategy: {strategy_name}")
    
    async def digest(self, 
                               start_url: str, 
                               query: str,
                               resume_from: Optional[str] = None) -> CrawlState:
        """Main entry point for adaptive crawling"""
        # Initialize or resume state
        if resume_from:
            self.state = CrawlState.load(resume_from)
            self.state.query = query  # Update query in case it changed
        else:
            self.state = CrawlState(
                crawled_urls=set(),
                knowledge_base=[],
                pending_links=[],
                query=query,
                metrics={}
            )

View on GitHub (pinned to 7e80152142)

Solutions

1. Reduce concurrent crawl load and let memory drain; retry once pressure drops below the threshold
2. Standardize browser_config across requests so pool reuse (no new browser) is possible
3. Raise the container memory limit or tune crawler.memory_threshold_percent if the threshold is set too close to steady-state usage
4. Restart the container / recycle idle browsers if leaked memory has permanently raised the baseline

Example fix

# server config: give headroom if steady-state sits near the threshold
# config
[crawler]
memory_threshold_percent = 90.0
# plus docker: --memory=4g instead of 2g

for attempt in range(3):
    try:
        crawler = await get_crawler(cfg)
        break
    except MemoryError:
        await asyncio.sleep(5 * (attempt + 1))  # let memory drain, then retry
else:
    shed_load()  # queue or reject the request upstream

When it happens

Trigger: Issuing crawl requests with a new/distinct browser_config signature (forcing a new browser) while container memory is >= the threshold; typically when existing browsers have leaked memory or concurrency is too high. Existing hot/cold pool browsers with matching signatures are still handed out.

Common situations: Many unique browser configs in one deployment fragmenting the pool; long-running container with bloated browsers; memory threshold lowered in config; small container memory limit vs page weight.

Related errors

• Memory usage exceeded threshold for {self.memory_wait_timeou
• Error evaluating condition: ${{error.message}}
• Invalid hook type: {hook_type}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/a595591342cff113. Report an issue: GitHub.