unclecode/crawl4ai · error · MemoryError

Memory at {mem_pct:.1f}%, refusing new browser

Error message

Memory at {mem_pct:.1f}%, refusing new browser

What it means

Error "Memory at {mem_pct:.1f}%, refusing new browser" thrown in unclecode/crawl4ai.

Source

Thrown at deploy/docker/crawler_pool.py:109

                HOT_POOL[sig] = COLD_POOL.pop(sig)

                # Track promotion in monitor
                try:
                    from monitor import get_monitor
                    await get_monitor().track_janitor_event("promote", sig, {"count": USAGE_COUNT[sig]})
                except:
                    pass

                return HOT_POOL[sig]

            logger.info(f"❄️  Using cold pool browser (sig={sig[:8]})")
            return crawler

        # Memory check before creating new
        mem_pct = get_container_memory_percent()
        if mem_pct >= MEM_LIMIT:
            logger.error(f"💥 Memory pressure: {mem_pct:.1f}% >= {MEM_LIMIT}%")
            raise MemoryError(f"Memory at {mem_pct:.1f}%, refusing new browser")

        # Create new in cold pool
        logger.info(f"🆕 Creating new browser in cold pool (sig={sig[:8]}, mem={mem_pct:.1f}%)")
        crawler = AsyncWebCrawler(config=cfg, thread_safe=False)
        await crawler.start()
        crawler.active_requests = 1
        COLD_POOL[sig] = crawler
        LAST_USED[sig] = time.time()
        USAGE_COUNT[sig] = 1
        return crawler

async def release_crawler(crawler: AsyncWebCrawler):
    """Decrement active request count for a pooled crawler.

    Call this in a finally block after finishing work with a crawler
    obtained via get_crawler() so the janitor knows when it's safe
    to close idle browsers.
    """

View on GitHub (pinned to 7e80152142)

Solutions

1. Wait for memory usage to drop or free memory on the host before requesting a new browser.
2. Lower the number of concurrent browsers or raise the memory threshold configuration if appropriate.

Example fix

# Retry after existing crawl jobs complete and memory decreases

When it happens

Trigger: Thrown at deploy/docker/crawler_pool.py:109 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/523bab73a97fa6c1. Report an issue: GitHub.