{"record":{"id":"60db273d3088368d","repo":"D4Vinci/Scrapling","slug":"failed-to-get-response-for-url","errorCode":null,"errorMessage":"Failed to get response for {url}","messagePattern":"Failed to get response for (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"scrapling/engines/_browsers/_controllers.py","lineNumber":168,"sourceCode":"                        page_info,\n                        final_response,\n                        xhr_pattern=self._config.capture_xhr,\n                        xhr_container=xhr_captured,\n                    ),\n                )\n\n                if params.page_setup:\n                    try:\n                        params.page_setup(page)\n                    except Exception as e:  # pragma: no cover\n                        log.error(f\"Error executing page_setup: {e}\")\n\n                try:\n                    first_response = page.goto(url, referer=referer)\n                    self._wait_for_page_stability(page, params.load_dom, params.network_idle)\n\n                    if not first_response:\n                        raise RuntimeError(f\"Failed to get response for {url}\")\n\n                    if params.page_action:\n                        try:\n                            _ = params.page_action(page)\n                        except Exception as e:  # pragma: no cover\n                            log.error(f\"Error executing page_action: {e}\")\n\n                    if params.wait_selector:\n                        try:\n                            waiter: Locator = page.locator(params.wait_selector)\n                            waiter.first.wait_for(state=params.wait_selector_state)\n                            self._wait_for_page_stability(page, params.load_dom, params.network_idle)\n                        except Exception as e:  # pragma: no cover\n                            log.error(f\"Error waiting for selector {params.wait_selector}: {e}\")\n\n                    page.wait_for_timeout(params.wait)\n\n                    response = ResponseFactory.from_playwright_response(","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/engines/_browsers/_controllers.py#L150-L186","documentation":"Raised in sync `DynamicSession.fetch`'s navigation step (scrapling/engines/_browsers/_controllers.py:168). `page.goto(url, referer=referer)` normally returns a `Response` object or `None`; `None` means Playwright navigated but could not associate a response (e.g. a download was triggered, navigation was intercepted/redirected in a way Playwright can't attribute, or the page went straight to an error state). Scrapling converts that `None` into an explicit RuntimeError naming the URL.","triggerScenarios":"The target URL starts a file download instead of rendering a page; a service worker or route handler intercepts the main request (`page.route` in `page_setup`); `about:blank` or `data:` style navigations; browser-provided responses lost on cross-origin redirect chains. Retryable in the fetch loop — it participates in the session's retry logic.","commonSituations":"Scraping links that sometimes serve files (PDF/CSV endpoints); a `page_setup` route that aborts or fulfills requests; sites that immediately redirect via HTTP 3xx chains Playwright reports oddly; headless Chromium version changes altering response attribution.","solutions":["Retry the fetch (the session does automatically up to `retries`) — transient attributions often succeed on retry.","If the URL is a download, use the HTTP-level fetcher (`Fetcher.get`) or Playwright's download events via `page_action` instead of `DynamicSession`.","Review any `page_setup` routes: don't abort/fulfill the main frame request.","Raise `retries`/add `retry_delay` for flaky redirect chains."],"exampleFix":"# before\nresp = session.fetch('https://example.com/report.csv')  # download -> RuntimeError\n\n# after\nfrom scrapling.fetchers import Fetcher\nresp = Fetcher.get('https://example.com/report.csv')  # HTTP client, no browser navigation","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    resp = session.fetch(url)\nexcept RuntimeError as e:\n    if 'Failed to get response' in str(e):\n        # likely a download or intercepted navigation; try HTTP client\n        from scrapling.fetchers import Fetcher\n        resp = Fetcher.get(url)\n    else:\n        raise","preventionTips":["Keep retries >= 2 so transient response-attribution failures self-heal.","Don't intercept the main-frame request in page_setup routes.","Use the HTTP fetcher for URLs that return files."],"tags":["browser","navigation","playwright","scrapling"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}