D4Vinci/Scrapling · error · SessionConfigurationError

Error in {self.__class__.__name__}.configure_sessions(): {e}

Error message

Error in {self.__class__.__name__}.configure_sessions(): {e}

What it means

A wrapper error: any exception raised inside your configure_sessions() override is caught and re-raised as SessionConfigurationError with the class name and original message, with the original exception chained via `from e`. The root cause is in the chained exception, not in scrapling.

Source

Thrown at scrapling/spiders/spider.py:148

        self.logger.addHandler(console_handler)

        if self.log_file:
            Path(self.log_file).parent.mkdir(parents=True, exist_ok=True)
            file_handler = logging.FileHandler(self.log_file)
            file_handler.setFormatter(formatter)
            self.logger.addHandler(file_handler)

        self.crawldir: Optional[Path] = Path(crawldir) if crawldir else None
        self._interval = interval
        self._engine: Optional[CrawlerEngine] = None
        self._original_sigint_handler: Any = None

        self._session_manager = SessionManager()

        try:
            self.configure_sessions(self._session_manager)
        except Exception as e:
            raise SessionConfigurationError(f"Error in {self.__class__.__name__}.configure_sessions(): {e}") from e

        if len(self._session_manager) == 0:
            raise SessionConfigurationError(f"{self.__class__.__name__}.configure_sessions() did not add any sessions")

        self.logger.info("Spider initialized")

    async def start_requests(self) -> AsyncGenerator[Request, None]:
        """Generate initial requests to start the crawl.

        By default, this generates Request objects for each URL in `start_urls`
        using the session manager's default session and `parse()` as callback.

        Override this method for more control over initial requests
        (e.g., to add custom headers, use different callbacks, etc.)
        """
        if not self.start_urls:
            raise RuntimeError(
                "Spider has no starting point, either set `start_urls` or override `start_requests` function."

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Read the chained original exception (`raise ... from e`) — the actual failure is in configure_sessions, not the wrapper
2. Reproduce the session construction standalone (e.g. build the FetcherSession outside the spider) to see the real error directly
3. Fix the constructor call / add() usage inside configure_sessions

Example fix

// before
def configure_sessions(self, manager):
    manager.add("default", FetcherSession(prxy="http://x"))  # typo'd kwarg raises

// after
def configure_sessions(self, manager):
    manager.add("default", FetcherSession(proxy="http://x"))

try:
    FetcherSession(proxy=proxy_url)  # prove session options are valid
except Exception:
    raise  # fix before the spider wraps it
manager.add("default", FetcherSession(proxy=proxy_url))

except SessionConfigurationError as e:
    cause = e.__cause__  # the real exception from configure_sessions
    log.error("session setup failed: %r", cause)

When it happens

Trigger: configure_sessions() raising while constructing sessions — invalid proxy strings, bad FetcherSession/AsyncDynamicSession constructor kwargs, a typo'd method call, or a network/credential error during session setup; manager.add() duplicate-id ValueError is also wrapped here.

Common situations: Typos in session constructor arguments; invalid user-agent or header configs; passing sync-only options to async sessions; duplicate session ids from a loop that reuses an id.

Related errors

• Session '{session_id}' already registered
• No sessions registered
• Session '{session_id}' not found. Available: {available}
• Session type {type(client)} not supported for async fetch
• No session found with the request session id

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/d2222730271d2b44. Report an issue: GitHub.