D4Vinci/Scrapling · error · RuntimeError

Failed to retrieve the page content after retrying for {max_

Error message

Failed to retrieve the page content after retrying for {max_retries * 500}ms.

What it means

Raised by ContentConverter._get_page_content (sync) after page.content() raised PlaywrightError on every one of max_retries (default 20) attempts, sleeping 500ms between tries — 10 seconds total by default. It exists as a workaround for Playwright issue #16108, where page.content() intermittently fails (classically on Windows); when the failure is persistent rather than transient (page closed, browser crashed, target destroyed), the retries exhaust and this RuntimeError is raised.

Source

Thrown at scrapling/engines/toolbelt/convertor.py:211

        except Exception as e:  # pragma: no cover
            log.error(f"Error processing response history: {e}")

        return history

    @classmethod
    def _get_page_content(cls, page: SyncPage, max_retries: int = 20) -> str:
        """
        A workaround for the Playwright issue with `page.content()` on Windows. Ref.: https://github.com/microsoft/playwright/issues/16108
        :param page: The page to extract content from.
        :param max_retries: Maximum number of retry attempts before raising `RuntimeError`.
        :return:
        """
        for _ in range(max_retries):
            try:
                return page.content() or ""
            except PlaywrightError:
                page.wait_for_timeout(500)
        raise RuntimeError(f"Failed to retrieve the page content after retrying for {max_retries * 500}ms.")

    @classmethod
    async def _get_async_page_content(cls, page: AsyncPage, max_retries: int = 20) -> str:
        """
        A workaround for the Playwright issue with `page.content()` on Windows. Ref.: https://github.com/microsoft/playwright/issues/16108
        :param page: The page to extract content from.
        :param max_retries: Maximum number of retry attempts before raising `RuntimeError`.
        :return:
        """
        for _ in range(max_retries):
            try:
                return (await page.content()) or ""
            except PlaywrightError:
                await page.wait_for_timeout(500)
        raise RuntimeError(f"Failed to retrieve the page content after retrying for {max_retries * 500}ms.")

    @classmethod
    async def from_async_playwright_response(

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Don't close the page/browser until after from_playwright_response has returned.
2. Catch PlaywrightError around the whole fetch and retry the full navigation instead of only content extraction.
3. Pass a larger max_retries only if you've confirmed the transient Windows bug; otherwise fix the underlying page instability (longer timeouts, wait_for_load_state before reading).

Example fix

# before
page.goto(url)
resp = Response.from_playwright_response(page, first, final)
page.close()  # if closed earlier in a timeout handler -> RuntimeError

# after
page.goto(url, wait_until='domcontentloaded')
page.wait_for_load_state('networkidle')
resp = Response.from_playwright_response(page, first, final)
page.close()  # close only after conversion completes

def page_is_readable(page) -> bool:
    try:
        return not page.is_closed()
    except Exception:
        return False

if not page_is_readable(page):
    raise RuntimeError('page closed before content extraction')

def page_is_readable(page) -> bool:
    try:
        return not page.is_closed()
    except Exception:
        return False

from playwright.sync_api import Error as PlaywrightError

for attempt in range(3):
    try:
        resp = Response.from_playwright_response(page, first, final)
        break
    except RuntimeError as e:
        if 'Failed to retrieve the page content' in str(e) and attempt < 2:
            page.wait_for_timeout(1000)
            continue
        raise

When it happens

Trigger: Calling page.close() (or the browser dying) while conversion reads content; navigating away mid-read so the execution context is destroyed; the transient Windows bug persisting longer than 10s; pages whose frame detaches repeatedly.

Common situations: Windows hosts scraping with the browser fetcher; racing a timeout that closes the page while the convertor runs; heavy pages where the renderer hangs; Playwright/browser version mismatches making page.content() systematically fail.

Related errors

• Failed to get a response from the page
• 'quality' is only valid when 'image_type' is 'jpeg'.
• Browser not initialized for proxy rotation mode
• Failed to get response for {url}
• Failed to get response for {url}

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/99be537a14fdb167. Report an issue: GitHub.