{"record":{"id":"b6148dc9686a9775","repo":"D4Vinci/Scrapling","slug":"maximum-page-limit-self-max-pages-reached","errorCode":null,"errorMessage":"Maximum page limit ({self.max_pages}) reached","messagePattern":"Maximum page limit \\((.+?)\\) reached","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"scrapling/engines/_browsers/_page.py","lineNumber":61,"sourceCode":"\n    __slots__ = (\"max_pages\", \"pages\", \"_lock\")\n\n    def __init__(self, max_pages: int = 5):\n        self.max_pages = max_pages\n        self.pages: List[PageInfo[SyncPage] | PageInfo[AsyncPage]] = []\n        self._lock = RLock()\n\n    @overload\n    def add_page(self, page: SyncPage) -> PageInfo[SyncPage]: ...\n\n    @overload\n    def add_page(self, page: AsyncPage) -> PageInfo[AsyncPage]: ...\n\n    def add_page(self, page: SyncPage | AsyncPage) -> PageInfo[SyncPage] | PageInfo[AsyncPage]:\n        \"\"\"Add a new page to the pool\"\"\"\n        with self._lock:\n            if len(self.pages) >= self.max_pages:\n                raise RuntimeError(f\"Maximum page limit ({self.max_pages}) reached\")\n\n            if isinstance(page, AsyncPage):\n                page_info: PageInfo[SyncPage] | PageInfo[AsyncPage] = cast(\n                    PageInfo[AsyncPage], PageInfo(page, \"ready\", \"\")\n                )\n            else:\n                page_info = cast(PageInfo[SyncPage], PageInfo(page, \"ready\", \"\"))\n\n            self.pages.append(page_info)\n            return page_info\n\n    @property\n    def pages_count(self) -> int:\n        \"\"\"Get the total number of pages\"\"\"\n        return len(self.pages)\n\n    @property\n    def busy_count(self) -> int:","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/engines/_browsers/_page.py#L43-L79","documentation":"Raised by `PagePool.add_page` (scrapling/engines/_browsers/_page.py:61) when trying to register a new browser page after the pool already holds `max_pages` entries. The pool caps concurrent pages per session; `add_page` is called internally when `_get_page` creates a page, so the RuntimeError surfaces when the pool is full — normally prevented upstream by the wait logic in `_base.py` that pauses until a page frees (see error 29) or by sizing `max_pages` to your workload.","triggerScenarios":"More pages created concurrently than `max_pages` in one session — e.g. `max_pages=3` while 5 threads/tasks fetch simultaneously and the pool-wait path is bypassed (explicit `context=` provided, so no wait), or direct `session.page_pool.add_page(page)` calls exceeding the cap.","commonSituations":"Thread/asyncio worker count above `max_pages`; batch jobs where each task opens a page without closing prior ones; forgetting that proxy-rotation mode creates extra contexts whose pages also enter the pool.","solutions":["Size `max_pages` at or above your maximum concurrency: `DynamicSession(max_pages=n_workers)`.","Ensure pages are released/closed promptly after each fetch so the pool drains.","Bound callers with a semaphore matching `max_pages`.","Catch RuntimeError around fetch and re-queue the URL instead of aborting the batch."],"exampleFix":"# before\nwith DynamicSession(max_pages=2) as s:\n    results = await asyncio.gather(*[s.fetch(u) for u in urls[:10]])  # >2 concurrent -> RuntimeError\n\n# after\nwith DynamicSession(max_pages=10) as s:\n    results = await asyncio.gather(*[s.fetch(u) for u in urls[:10]])","handlingStrategy":"validation","validationCode":"def page_pool_has_capacity(session, extra: int = 1) -> bool:\n    pool = session.page_pool\n    return pool.pages_count + extra <= pool.max_pages","typeGuard":null,"tryCatchPattern":"try:\n    resp = session.fetch(url)\nexcept RuntimeError as e:\n    if 'Maximum page limit' in str(e):\n        await asyncio.sleep(1)  # let pages free up, then retry\n        resp = session.fetch(url)\n    else:\n        raise","preventionTips":["Set max_pages at session creation to your max concurrency.","Close/release pages promptly after each fetch.","Cap callers with a semaphore sized to max_pages."],"tags":["browser","page-pool","concurrency","scrapling"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}