D4Vinci/Scrapling · error · RuntimeError

Maximum page limit ({self.max_pages}) reached

Error message

Maximum page limit ({self.max_pages}) reached

What it means

Raised by `PagePool.add_page` (scrapling/engines/_browsers/_page.py:61) when trying to register a new browser page after the pool already holds `max_pages` entries. The pool caps concurrent pages per session; `add_page` is called internally when `_get_page` creates a page, so the RuntimeError surfaces when the pool is full — normally prevented upstream by the wait logic in `_base.py` that pauses until a page frees (see error 29) or by sizing `max_pages` to your workload.

Source

Thrown at scrapling/engines/_browsers/_page.py:61

    __slots__ = ("max_pages", "pages", "_lock")

    def __init__(self, max_pages: int = 5):
        self.max_pages = max_pages
        self.pages: List[PageInfo[SyncPage] | PageInfo[AsyncPage]] = []
        self._lock = RLock()

    @overload
    def add_page(self, page: SyncPage) -> PageInfo[SyncPage]: ...

    @overload
    def add_page(self, page: AsyncPage) -> PageInfo[AsyncPage]: ...

    def add_page(self, page: SyncPage | AsyncPage) -> PageInfo[SyncPage] | PageInfo[AsyncPage]:
        """Add a new page to the pool"""
        with self._lock:
            if len(self.pages) >= self.max_pages:
                raise RuntimeError(f"Maximum page limit ({self.max_pages}) reached")

            if isinstance(page, AsyncPage):
                page_info: PageInfo[SyncPage] | PageInfo[AsyncPage] = cast(
                    PageInfo[AsyncPage], PageInfo(page, "ready", "")
                )
            else:
                page_info = cast(PageInfo[SyncPage], PageInfo(page, "ready", ""))

            self.pages.append(page_info)
            return page_info

    @property
    def pages_count(self) -> int:
        """Get the total number of pages"""
        return len(self.pages)

    @property
    def busy_count(self) -> int:

View on GitHub (pinned to 5d213a2d47)

Solutions

  1. Size `max_pages` at or above your maximum concurrency: `DynamicSession(max_pages=n_workers)`.
  2. Ensure pages are released/closed promptly after each fetch so the pool drains.
  3. Bound callers with a semaphore matching `max_pages`.
  4. Catch RuntimeError around fetch and re-queue the URL instead of aborting the batch.

Example fix

# before
with DynamicSession(max_pages=2) as s:
    results = await asyncio.gather(*[s.fetch(u) for u in urls[:10]])  # >2 concurrent -> RuntimeError

# after
with DynamicSession(max_pages=10) as s:
    results = await asyncio.gather(*[s.fetch(u) for u in urls[:10]])
Defensive patterns

Strategy: validation

Validate before calling

def page_pool_has_capacity(session, extra: int = 1) -> bool:
    pool = session.page_pool
    return pool.pages_count + extra <= pool.max_pages

Try / catch

try:
    resp = session.fetch(url)
except RuntimeError as e:
    if 'Maximum page limit' in str(e):
        await asyncio.sleep(1)  # let pages free up, then retry
        resp = session.fetch(url)
    else:
        raise

Prevention

When it happens

Trigger: More pages created concurrently than `max_pages` in one session — e.g. `max_pages=3` while 5 threads/tasks fetch simultaneously and the pool-wait path is bypassed (explicit `context=` provided, so no wait), or direct `session.page_pool.add_page(page)` calls exceeding the cap.

Common situations: Thread/asyncio worker count above `max_pages`; batch jobs where each task opens a page without closing prior ones; forgetting that proxy-rotation mode creates extra contexts whose pages also enter the pool.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/b6148dc9686a9775. Report an issue: GitHub.