{"record":{"id":"b0794165e941c57e","repo":"unclecode/crawl4ai","slug":"memory-usage-exceeded-threshold-for-self-memory-w","errorCode":null,"errorMessage":"Memory usage exceeded threshold for {self.memory_wait_timeout} seconds","messagePattern":"Memory usage exceeded threshold for (.+?) seconds","errorType":"exception","errorClass":"MemoryError","httpStatus":null,"severity":"critical","filePath":"crawl4ai/async_dispatcher.py","lineNumber":195,"sourceCode":"        while True:\n            self.current_memory_percent = get_true_memory_usage_percent()\n\n            # Enter memory pressure mode if we cross the threshold\n            if self.current_memory_percent >= self.memory_threshold_percent:\n                if not self.memory_pressure_mode:\n                    self.memory_pressure_mode = True\n                    self._high_memory_start_time = time.time()\n                    if self.monitor:\n                        self.monitor.update_memory_status(\"PRESSURE\")\n                else:\n                    if self._high_memory_start_time is None:\n                        self._high_memory_start_time = time.time()\n                    if (\n                        self.memory_wait_timeout is not None\n                        and self._high_memory_start_time is not None\n                        and time.time() - self._high_memory_start_time >= self.memory_wait_timeout\n                    ):\n                        raise MemoryError(\n                            \"Memory usage exceeded threshold for\"\n                            f\" {self.memory_wait_timeout} seconds\"\n                        )\n\n            # Exit memory pressure mode if we go below recovery threshold\n            elif self.memory_pressure_mode and self.current_memory_percent <= self.recovery_threshold_percent:\n                self.memory_pressure_mode = False\n                self._high_memory_start_time = None\n                if self.monitor:\n                    self.monitor.update_memory_status(\"NORMAL\")\n            elif self.current_memory_percent < self.memory_threshold_percent:\n                self._high_memory_start_time = None\n            \n            # In critical mode, we might need to take more drastic action\n            if self.current_memory_percent >= self.critical_threshold_percent:\n                if self.monitor:\n                    self.monitor.update_memory_status(\"CRITICAL\")\n                # We could implement additional memory-saving measures here","sourceCodeStart":177,"sourceCodeEnd":213,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_dispatcher.py#L177-L213","documentation":"Raised by the memory monitor in the async dispatcher (SemaphoreDispatcher / MemoryAwareDispatcher) when system memory usage stays above memory_threshold_percent continuously for longer than memory_wait_timeout seconds. It is a hard abort to prevent OOM kills of the host process. The timestamp is tracked via _high_memory_start_time, which resets only when memory falls below the recovery/threshold level.","triggerScenarios":"Running arun_many() with a large URL batch on a memory-constrained host (or in a small Docker container) while memory stays above memory_threshold_percent (default ~70%) for memory_wait_timeout seconds. Heavy parallel page loads, huge HTML payloads, or an actual system-wide memory leak from another process can also push the percentage over the line.","commonSituations":"Docker/Kubernetes containers with low memory limits; crawling sites with very large pages; running many concurrent crawls on the same box; system-level memory pressure from other services; dispatcher defaults too aggressive for the machine.","solutions":["Free or add system memory (or raise the container memory limit) and rerun the batch.","Reduce crawler concurrency: lower CrawlerRunConfig-like concurrency settings, e.g. dispatcher rate_limiter or max concurrent tasks via MemoryAwareDispatcher(..., max_session_per_inst, memory_threshold_percent=85).","Raise memory_threshold_percent and/or memory_wait_timeout in the dispatcher config so transient pressure does not abort the run.","Enable/verify check_interval and recovery_threshold_percent so the monitor exits pressure mode before the timeout fires.","Split the URL list into smaller batches and restart between them; or free other memory-hungry processes on the host."],"exampleFix":"# before\ndispatcher = MemoryAwareDispatcher(rate_limiter=rl)  # defaults abort quickly under pressure\n\n# after\ndispatcher = MemoryAwareDispatcher(\n    rate_limiter=rl,\n    memory_threshold_percent=85,   # tolerate more pressure\n    memory_wait_timeout=60,          # wait longer before aborting\n)","handlingStrategy":"retry","validationCode":"import psutil\nmem = psutil.virtual_memory().percent\nif mem > 80:\n    raise SystemExit(f'system memory at {mem}% - free memory before crawling')","typeGuard":null,"tryCatchPattern":"try:\n    results = await dispatcher.run_urls(urls)\nexcept MemoryError:\n    # abort batch, free memory, optionally restart with lower concurrency\n    await dispatcher.cleanup()\n    raise","preventionTips":["Set dispatcher memory_threshold_percent and memory_wait_timeout to match host capacity.","Run crawl batches against psutil.virtual_memory() headroom checks before starting.","Cap concurrency and page sizes (e.g. text_mode=True) on small containers.","Monitor with the built-in monitor (update_memory_status) and alert on PRESSURE."],"tags":["memory","dispatcher","resource-limits","async"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}