D4Vinci/Scrapling · error · RuntimeError

Failed to get response for {url}

Error message

Failed to get response for {url}

What it means

Raised when Playwright's page.goto(url) completes but returns None, meaning no main network response backs the navigation. This happens for navigations to non-HTTP targets (about:blank, file downloads, javascript: or data: redirects) or when the navigation is intercepted/cancelled, so scrapling refuses to fabricate a Response.

Source

Thrown at scrapling/engines/_browsers/_stealth.py:251

                        page_info,
                        final_response,
                        xhr_pattern=self._config.capture_xhr,
                        xhr_container=xhr_captured,
                    ),
                )

                if params.page_setup:
                    try:
                        params.page_setup(page)
                    except Exception as e:  # pragma: no cover
                        log.error(f"Error executing page_setup: {e}")

                try:
                    first_response = page.goto(url, referer=referer)
                    self._wait_for_page_stability(page, params.load_dom, params.network_idle)

                    if not first_response:
                        raise RuntimeError(f"Failed to get response for {url}")

                    if params.solve_cloudflare:
                        self._cloudflare_solver(page)
                        # Make sure the page is fully loaded after the captcha
                        self._wait_for_page_stability(page, params.load_dom, params.network_idle)

                    if params.page_action:
                        try:
                            _ = params.page_action(page)
                        except Exception as e:  # pragma: no cover
                            log.error(f"Error executing page_action: {e}")

                    if params.wait_selector:
                        try:
                            waiter: Locator = page.locator(params.wait_selector)
                            waiter.first.wait_for(state=params.wait_selector_state)
                            self._wait_for_page_stability(page, params.load_dom, params.network_idle)
                        except Exception as e:  # pragma: no cover

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Verify the URL actually returns an HTML/document response with a plain HTTP client (e.g. requests) before sending it to the browser
2. Disable block_ads/blocked_domains entries that may match the target domain and retry
3. If the target serves downloads, handle it outside the browser session or expect this error per-URL

Example fix

// before
resp = session.fetch('https://example.com/report.pdf')  # goto -> None

// after
if not url.lower().endswith(('.pdf','.zip','.csv')):
    resp = session.fetch(url)

import re
DOCUMENT_LIKE = re.compile(r'^https?://', re.I)
def safe_url(u: str) -> bool:
    return bool(DOCUMENT_LIKE.match(u)) and not u.lower().endswith(('.pdf','.zip','.csv','.exe','.dmg'))

def is_navigable_document(url: str) -> bool:
    return url.startswith(('http://', 'https://')) and not re.search(r'\.(pdf|zip|csv|tar|gz|exe)$', url, re.I)

for url in urls:
    try:
        resp = session.fetch(url)
    except RuntimeError as e:
        if 'Failed to get response' in str(e):
            log.warning('skipping non-document url %s', url); continue
        raise

When it happens

Trigger: Navigating to a URL that triggers a download, redirects to about:blank or a chrome:// page, or a URL whose main request is blocked (e.g. by an interceptor, blocked_domains, or a failing proxy) so no primary response object is produced.

Common situations: Scraping links extracted from pages that turn out to be direct file downloads, proxy interception swallowing the main request, or ad/domain blockers cancelling navigation to the target.

Related errors

• Failed to get response for {url}
• 'quality' is only valid when 'image_type' is 'jpeg'.
• Browser not initialized for proxy rotation mode
• Session has been already started
• Context manager has been closed

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/dec3b0a31d26e4b9. Report an issue: GitHub.