What causes "Failed to get response for {url}" in D4Vinci/Scrapling?

Retry the fetch (the session does automatically up to `retries`) — transient attributions often succeed on retry.

D4Vinci/Scrapling · error · RuntimeError

Failed to get response for {url}

Error message

Failed to get response for {url}

What it means

Raised in sync `DynamicSession.fetch`'s navigation step (scrapling/engines/_browsers/_controllers.py:168). `page.goto(url, referer=referer)` normally returns a `Response` object or `None`; `None` means Playwright navigated but could not associate a response (e.g. a download was triggered, navigation was intercepted/redirected in a way Playwright can't attribute, or the page went straight to an error state). Scrapling converts that `None` into an explicit RuntimeError naming the URL.

Source

Thrown at scrapling/engines/_browsers/_controllers.py:168

                        page_info,
                        final_response,
                        xhr_pattern=self._config.capture_xhr,
                        xhr_container=xhr_captured,
                    ),
                )

                if params.page_setup:
                    try:
                        params.page_setup(page)
                    except Exception as e:  # pragma: no cover
                        log.error(f"Error executing page_setup: {e}")

                try:
                    first_response = page.goto(url, referer=referer)
                    self._wait_for_page_stability(page, params.load_dom, params.network_idle)

                    if not first_response:
                        raise RuntimeError(f"Failed to get response for {url}")

                    if params.page_action:
                        try:
                            _ = params.page_action(page)
                        except Exception as e:  # pragma: no cover
                            log.error(f"Error executing page_action: {e}")

                    if params.wait_selector:
                        try:
                            waiter: Locator = page.locator(params.wait_selector)
                            waiter.first.wait_for(state=params.wait_selector_state)
                            self._wait_for_page_stability(page, params.load_dom, params.network_idle)
                        except Exception as e:  # pragma: no cover
                            log.error(f"Error waiting for selector {params.wait_selector}: {e}")

                    page.wait_for_timeout(params.wait)

                    response = ResponseFactory.from_playwright_response(

View on GitHub (pinned to 5d213a2d47)

Solutions

Retry the fetch (the session does automatically up to `retries`) — transient attributions often succeed on retry.
If the URL is a download, use the HTTP-level fetcher (`Fetcher.get`) or Playwright's download events via `page_action` instead of `DynamicSession`.
Review any `page_setup` routes: don't abort/fulfill the main frame request.
Raise `retries`/add `retry_delay` for flaky redirect chains.

Example fix

# before
resp = session.fetch('https://example.com/report.csv')  # download -> RuntimeError

# after
from scrapling.fetchers import Fetcher
resp = Fetcher.get('https://example.com/report.csv')  # HTTP client, no browser navigation

Defensive patterns

Strategy: retry

Try / catch

try:
    resp = session.fetch(url)
except RuntimeError as e:
    if 'Failed to get response' in str(e):
        # likely a download or intercepted navigation; try HTTP client
        from scrapling.fetchers import Fetcher
        resp = Fetcher.get(url)
    else:
        raise

Prevention

Keep retries >= 2 so transient response-attribution failures self-heal.
Don't intercept the main-frame request in page_setup routes.
Use the HTTP fetcher for URLs that return files.

When it happens

Trigger: The target URL starts a file download instead of rendering a page; a service worker or route handler intercepts the main request (`page.route` in `page_setup`); `about:blank` or `data:` style navigations; browser-provided responses lost on cross-origin redirect chains. Retryable in the fetch loop — it participates in the session's retry logic.

Common situations: Scraping links that sometimes serve files (PDF/CSV endpoints); a `page_setup` route that aborts or fulfills requests; sites that immediately redirect via HTTP 3xx chains Playwright reports oddly; headless Chromium version changes altering response attribution.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/60db273d3088368d. Report an issue: GitHub.