D4Vinci/Scrapling · error · ValueError

Failed to get a response from the page

Error message

Failed to get a response from the page

What it means

Raised by Response.from_playwright_response (sync flavor) when both final_response and first_response are falsy. The convertor needs at least one Playwright Response object to read status, headers, and body from; if navigation produced no capturable document response (or the caller passed None for both), conversion is impossible, so it raises ValueError immediately.

Source

Thrown at scrapling/engines/toolbelt/convertor.py:115

        by falling back to the first response if necessary. Encoding and status text
        are also derived from the provided response headers or reasonable defaults.
        Additionally, the page content and cookies are extracted for further use.

        :param page: A synchronous Playwright `Page` instance that represents the current browser page. Required to retrieve the page's URL, cookies, and content.
        :param final_response: The last response received for the given request from the Playwright instance. Typically used as the main response object to derive status, headers, and other metadata.
        :param first_response: An earlier or initial Playwright `Response` object that may serve as a fallback response in the absence of the final one.
        :param parser_arguments: A dictionary containing additional arguments needed for parsing or further customization of the returned `Response`. These arguments are dynamically unpacked into
            the `Response` object.
        :param meta: Additional meta data to be saved with the response.
        :param xhr_captured: Optional list of captured Playwright XHR/fetch responses to convert and attach to the returned Response.
        :param collect_history: Optional boolean indicating whether to collect redirections history or not.
        :return: A fully populated `Response` object containing the page's URL, content, status, headers, cookies, and other derived metadata.
        :rtype: Response
        """
        # In case we didn't catch a document type somehow
        final_response = final_response if final_response else first_response
        if not final_response:
            raise ValueError("Failed to get a response from the page")

        encoding = cls.__extract_browser_encoding(final_response.headers.get("content-type", ""))
        # PlayWright API sometimes give empty status text for some reason!
        status_text = final_response.status_text or StatusText.get(final_response.status)

        history = cls._process_response_history(first_response, parser_arguments) if collect_history else []
        try:
            if page and "html" in final_response.all_headers().get("content-type", ""):
                page_content = cls._get_page_content(page).encode("utf-8")
                encoding = "utf-8"
            else:
                page_content = final_response.body()
        except Exception as e:  # pragma: no cover
            log.error(f"Error getting page content: {e}")
            page_content = b""

        response = Response(
            **{

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Guarantee the caller keeps at least the response returned by page.goto() and passes it as first_response/final_response.
2. Attach response listeners before triggering navigation so the initial redirect response is captured.
3. If the response may legitimately be absent (blank pages, canceled navigations), catch ValueError and skip/retry the page instead of converting.

Example fix

# before
resp = Response.from_playwright_response(page, first_response=None, final_response=None)  # ValueError

# after
pw_resp = page.goto(url)  # may be None on some navigations
if pw_resp is None:
    pw_resp = next((r for r in captured if r.request.is_navigation_request()), None)
if pw_resp is None:
    raise RuntimeError(f'No document response captured for {url}')
resp = Response.from_playwright_response(page, first_response=captured[0], final_response=pw_resp)

def convert_page(page, captured):
    final = next((r for r in captured if r.request.is_navigation_request() and r.redirected_to is None), None) or (captured[-1] if captured else None)
    first = captured[0] if captured else None
    if final is None and first is None:
        raise RuntimeError(f'No document response captured for {page.url}')
    return Response.from_playwright_response(page, first, final)

def has_any_response(*responses) -> bool:
    return any(r is not None for r in responses)

try:
    resp = Response.from_playwright_response(page, first, final)
except ValueError as e:
    if 'Failed to get a response' in str(e):
        logger.warning('no document response for %s, skipping', page.url)
        return None
    raise

When it happens

Trigger: Calling from_playwright_response(first_response=None, final_response=None); page.goto() to a URL that never fires a document response (about:blank, failed DNS before response, service-worker-served pages); response capture filters (e.g., only XHR) skipping the document; waiting with 'domcontentloaded' plus redirects losing the tracked response.

Common situations: Custom fetch logic built on StealthyFetcher/Playwright where the response listener misses the main document (race between listener attach and goto, or page opened via page.content() without navigation); handling non-HTTP schemes; upstream Playwright behavior changes dropping the response object.

Related errors

• 'quality' is only valid when 'image_type' is 'jpeg'.
• Failed to retrieve the page content after retrying for {max_
• Invalid proxy string!
• Invalid proxy dictionary: {e}
• Argument `selector_config` must be a dictionary.

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/705fe62c9e357161. Report an issue: GitHub.