What causes "Request failed" in D4Vinci/Scrapling?

Set `retries >= 1` (default) so the final attempt re-raises the original, more informative exception instead of falling through.

D4Vinci/Scrapling · error · RuntimeError

Request failed

Error message

Request failed

What it means

End-of-function raise in sync `DynamicSession.fetch` (scrapling/engines/_browsers/_controllers.py:212), reached only when the retry loop finishes without returning and without re-raising — practically it fires when `self._config.retries` is 0 and the single attempt failed without raising (or the loop exits through a path that neither returns nor raises). It is the 'impossible state' backstop, marked `# pragma: no cover`; in almost all real failures the last exception is re-raised with `raise` inside the loop instead.

Source

Thrown at scrapling/engines/_browsers/_controllers.py:212

                    return response

                except Exception as e:
                    page_info.mark_error()
                    if attempt < self._config.retries - 1:
                        if is_proxy_error(e):
                            log.warning(
                                f"Proxy '{proxy}' failed (attempt {attempt + 1}) | Retrying in {self._config.retry_delay}s..."
                            )
                        else:
                            log.warning(
                                f"Attempt {attempt + 1} failed: {e}. Retrying in {self._config.retry_delay}s..."
                            )
                        time_sleep(self._config.retry_delay)
                    else:
                        log.error(f"Failed after {self._config.retries} attempts: {e}")
                        raise

        raise RuntimeError("Request failed")  # pragma: no cover


class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
    """An async Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory."""

    __slots__ = (
        "_config",
        "_context_options",
        "_browser_options",
        "_user_data_dir",
        "_headers_keys",
    )

    def __init__(self, **kwargs: Unpack[PlaywrightSession]):
        """A Browser session manager with page pooling

        :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
        :param disable_resources: Drop requests for unnecessary resources for a speed boost.

View on GitHub (pinned to 5d213a2d47)

Solutions

Set `retries >= 1` (default) so the final attempt re-raises the original, more informative exception instead of falling through.
Don't swallow exceptions in `page_action`/`page_setup` — let them propagate so the retry logic sees the real error.
Log `params` and the URL when this fires to reconstruct which attempt path fell through.
Report as a bug if it reproduces with default `retries`, since the loop should always return or re-raise.

Example fix

# before
session = DynamicSession(PlaywrightConfig(retries=0))
resp = session.fetch(url)  # falls through -> generic 'Request failed'

# after
session = DynamicSession(PlaywrightConfig(retries=2, retry_delay=1))
resp = session.fetch(url)  # last attempt re-raises the real exception

Defensive patterns

Strategy: retry

Validate before calling

def retries_sane(config) -> bool:
    return getattr(config, 'retries', 0) >= 1

Try / catch

try:
    resp = session.fetch(url)
except RuntimeError as e:
    if str(e) == 'Request failed':
        log.error('fetch fell through retry loop; check retries>=1 and page_action exception handling')
    raise

Prevention

Never set PlaywrightConfig(retries=0) unless you handle all failures yourself.
Don't swallow exceptions inside page_action/page_setup.
Log the final attempt's exception rather than relying on this generic message.

When it happens

Trigger: Constructing a session with `retries=0` and hitting a navigation that returns no response without another exception; code that catches exceptions inside `page_action`/`page_setup` swallowing the failure so the loop simply exhausts; theoretically unreachable when `retries >= 1` and the last attempt raises.

Common situations: Setting `retries=0` in `PlaywrightConfig` to disable retries and then seeing this generic message instead of the underlying cause; user callbacks that swallow exceptions making the loop fall through.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/a5e0321696fa2a4b. Report an issue: GitHub.