unclecode/crawl4ai · critical · MemoryError

Memory usage exceeded threshold for {self.memory_wait_timeou

Error message

Memory usage exceeded threshold for {self.memory_wait_timeout} seconds

What it means

Raised by the memory monitor in the async dispatcher (SemaphoreDispatcher / MemoryAwareDispatcher) when system memory usage stays above memory_threshold_percent continuously for longer than memory_wait_timeout seconds. It is a hard abort to prevent OOM kills of the host process. The timestamp is tracked via _high_memory_start_time, which resets only when memory falls below the recovery/threshold level.

Source

Thrown at crawl4ai/async_dispatcher.py:195

        while True:
            self.current_memory_percent = get_true_memory_usage_percent()

            # Enter memory pressure mode if we cross the threshold
            if self.current_memory_percent >= self.memory_threshold_percent:
                if not self.memory_pressure_mode:
                    self.memory_pressure_mode = True
                    self._high_memory_start_time = time.time()
                    if self.monitor:
                        self.monitor.update_memory_status("PRESSURE")
                else:
                    if self._high_memory_start_time is None:
                        self._high_memory_start_time = time.time()
                    if (
                        self.memory_wait_timeout is not None
                        and self._high_memory_start_time is not None
                        and time.time() - self._high_memory_start_time >= self.memory_wait_timeout
                    ):
                        raise MemoryError(
                            "Memory usage exceeded threshold for"
                            f" {self.memory_wait_timeout} seconds"
                        )

            # Exit memory pressure mode if we go below recovery threshold
            elif self.memory_pressure_mode and self.current_memory_percent <= self.recovery_threshold_percent:
                self.memory_pressure_mode = False
                self._high_memory_start_time = None
                if self.monitor:
                    self.monitor.update_memory_status("NORMAL")
            elif self.current_memory_percent < self.memory_threshold_percent:
                self._high_memory_start_time = None
            
            # In critical mode, we might need to take more drastic action
            if self.current_memory_percent >= self.critical_threshold_percent:
                if self.monitor:
                    self.monitor.update_memory_status("CRITICAL")
                # We could implement additional memory-saving measures here

View on GitHub (pinned to 7e80152142)

Solutions

1. Free or add system memory (or raise the container memory limit) and rerun the batch.
2. Reduce crawler concurrency: lower CrawlerRunConfig-like concurrency settings, e.g. dispatcher rate_limiter or max concurrent tasks via MemoryAwareDispatcher(..., max_session_per_inst, memory_threshold_percent=85).
3. Raise memory_threshold_percent and/or memory_wait_timeout in the dispatcher config so transient pressure does not abort the run.
4. Enable/verify check_interval and recovery_threshold_percent so the monitor exits pressure mode before the timeout fires.
5. Split the URL list into smaller batches and restart between them; or free other memory-hungry processes on the host.

Example fix

# before
dispatcher = MemoryAwareDispatcher(rate_limiter=rl)  # defaults abort quickly under pressure

# after
dispatcher = MemoryAwareDispatcher(
    rate_limiter=rl,
    memory_threshold_percent=85,   # tolerate more pressure
    memory_wait_timeout=60,          # wait longer before aborting
)

import psutil
mem = psutil.virtual_memory().percent
if mem > 80:
    raise SystemExit(f'system memory at {mem}% - free memory before crawling')

try:
    results = await dispatcher.run_urls(urls)
except MemoryError:
    # abort batch, free memory, optionally restart with lower concurrency
    await dispatcher.cleanup()
    raise

When it happens

Trigger: Running arun_many() with a large URL batch on a memory-constrained host (or in a small Docker container) while memory stays above memory_threshold_percent (default ~70%) for memory_wait_timeout seconds. Heavy parallel page loads, huge HTML payloads, or an actual system-wide memory leak from another process can also push the percentage over the line.

Common situations: Docker/Kubernetes containers with low memory limits; crawling sites with very large pages; running many concurrent crawls on the same box; system-level memory pressure from other services; dispatcher defaults too aggressive for the machine.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Unknown strategy: {strategy_name}
• Error evaluating condition: ${{error.message}}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/b0794165e941c57e. Report an issue: GitHub.