unclecode/crawl4ai · error · MemoryError
Memory at %, refusing new browser
Error message
Memory at {mem_pct:.1f}%, refusing new browser What it means
Error "Memory at {mem_pct:.1f}%, refusing new browser" thrown in unclecode/crawl4ai.
Solutions
- Wait for memory usage to drop or free memory on the host before requesting a new browser.
- Lower the number of concurrent browsers or raise the memory threshold configuration if appropriate.
Example fix
# Retry after existing crawl jobs complete and memory decreases
When it happens
Trigger: Thrown at deploy/docker/crawler_pool.py:109 when the library encounters an invalid state.
Common situations: See trigger scenarios.
AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14).
Data as JSON: /api/errors/523bab73a97fa6c1.
Report an issue: GitHub.
Appendix: source
Thrown at deploy/docker/crawler_pool.py:109
HOT_POOL[sig] = COLD_POOL.pop(sig)
# Track promotion in monitor
try:
from monitor import get_monitor
await get_monitor().track_janitor_event("promote", sig, {"count": USAGE_COUNT[sig]})
except:
pass
return HOT_POOL[sig]
logger.info(f"❄️ Using cold pool browser (sig={sig[:8]})")
return crawler
# Memory check before creating new
mem_pct = get_container_memory_percent()
if mem_pct >= MEM_LIMIT:
logger.error(f"💥 Memory pressure: {mem_pct:.1f}% >= {MEM_LIMIT}%")
raise MemoryError(f"Memory at {mem_pct:.1f}%, refusing new browser")
# Create new in cold pool
logger.info(f"🆕 Creating new browser in cold pool (sig={sig[:8]}, mem={mem_pct:.1f}%)")
crawler = AsyncWebCrawler(config=cfg, thread_safe=False)
await crawler.start()
crawler.active_requests = 1
COLD_POOL[sig] = crawler
LAST_USED[sig] = time.time()
USAGE_COUNT[sig] = 1
return crawler
async def release_crawler(crawler: AsyncWebCrawler):
"""Decrement active request count for a pooled crawler.
Call this in a finally block after finishing work with a crawler
obtained via get_crawler() so the janitor knows when it's safe
to close idle browsers.
"""View on GitHub (pinned to 7e80152142)