What causes "Context manager has been closed" in D4Vinci/Scrapling?

Move every `fetch` inside the `with DynamicSession() as session:` block.

D4Vinci/Scrapling · error · RuntimeError

Context manager has been closed

Error message

Context manager has been closed

What it means

Raised by sync `DynamicSession.fetch` (scrapling/engines/_browsers/_controllers.py:127) when `self._is_alive` is False — i.e. `fetch()` is called outside the session's `with` block or after it exited. The flag is set True in `__enter__` and False in `__exit__`, so this guards against fetching with a torn-down Playwright instance. Marked `no cover` because normal usage inside the context manager never sees it.

Source

Thrown at scrapling/engines/_browsers/_controllers.py:127

        :param page_action: Added for automation. A function that takes the `page` object, runs after navigation, and does the automation you need.
        :param page_setup: A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads.
        :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._
        :param disable_resources: Drop requests for unnecessary resources for a speed boost.
            Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
        :param blocked_domains: A set of domain names to block requests to. Subdomains are also matched (e.g., ``"example.com"`` blocks ``"sub.example.com"`` too).
        :param wait_selector: Wait for a specific CSS selector to be in a specific state.
        :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
        :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
        :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
        :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
        :param proxy: Static proxy to override rotator and session proxy. A new browser context will be created and used with it.
        :return: A `Response` object.
        """
        static_proxy = kwargs.pop("proxy", None)

        params = _validate(kwargs, self, PlaywrightConfig)
        if not self._is_alive:  # pragma: no cover
            raise RuntimeError("Context manager has been closed")

        request_headers_keys = {h.lower() for h in params.extra_headers.keys()} if params.extra_headers else set()
        referer = (
            "https://www.google.com/" if (params.google_search and "referer" not in request_headers_keys) else None
        )

        for attempt in range(self._config.retries):
            proxy: Optional[ProxyType] = None
            if self._config.proxy_rotator and static_proxy is None:
                proxy = self._config.proxy_rotator.get_proxy()
            else:
                proxy = static_proxy

            with self._page_generator(
                params.timeout, params.extra_headers, params.disable_resources, proxy, params.blocked_domains
            ) as page_info:
                final_response: List = [None]
                xhr_captured: List = []

View on GitHub (pinned to 5d213a2d47)

Solutions

Move every `fetch` inside the `with DynamicSession() as session:` block.
If fetches happen lazily, restructure so the context manager wraps the whole workload (queue + worker).
For one-off fetches, use the function-style API (e.g. `fetcher` / `DynamicSession` alternatives like `scrapling.fetchers`) that manages lifecycle per call.
Add a liveness assert before fetch in debug builds to catch misuse early.

Example fix

# before
with DynamicSession() as session:
    pass
resp = session.fetch(url)  # RuntimeError: closed

# after
with DynamicSession() as session:
    resp = session.fetch(url)

Defensive patterns

Strategy: type-guard

Validate before calling

def is_session_alive(session) -> bool:
    return bool(getattr(session, '_is_alive', False))

Type guard

def session_can_fetch(session) -> bool:
    """True while the session's context manager is active."""
    return bool(getattr(session, '_is_alive', False))

Try / catch

try:
    resp = session.fetch(url)
except RuntimeError as e:
    if 'closed' in str(e):
        with DynamicSession() as fresh:  # restart and retry once
            resp = fresh.fetch(url)
    else:
        raise

Prevention

Keep all fetch calls inside the `with` block.
Don't let queued/deferred work outlive the context manager.
Use one-shot fetchers for single requests instead of session objects.

When it happens

Trigger: `s = DynamicSession(); s.fetch(url)` (no `with`), or `with DynamicSession() as s: ...` followed by `s.fetch(url)` after the block; also fetching after `__exit__` ran because an earlier exception unwound the block.

Common situations: Refactoring code out of a `with` block and forgetting the session dies with it; storing the session globally and calling fetch lazily (e.g. in a callback that fires later); exception paths that exit the context while queued work still runs.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/f3b81463bf8d5ad6. Report an issue: GitHub.