What causes "{self.__class__.__name__} must implement parse() method" in D4Vinci/Scrapling?

Implement `async def parse(self, response)` as an async generator yielding Dict items, Requests, or None

D4Vinci/Scrapling · error · NotImplementedError

{self.class.name} must implement parse() method

Error message

{self.__class__.__name__} must implement parse() method

What it means

Spider.parse is an abstract method; every concrete spider must implement it because it is the default callback for responses whose Request has no explicit callback. The base implementation raises NotImplementedError when called.

Source

Thrown at scrapling/spiders/spider.py:175

        By default, this generates Request objects for each URL in `start_urls`
        using the session manager's default session and `parse()` as callback.

        Override this method for more control over initial requests
        (e.g., to add custom headers, use different callbacks, etc.)
        """
        if not self.start_urls:
            raise RuntimeError(
                "Spider has no starting point, either set `start_urls` or override `start_requests` function."
            )

        for url in self.start_urls:
            yield Request(url, sid=self._session_manager.default_session_id)

    @abstractmethod
    async def parse(self, response: "Response") -> AsyncGenerator[Dict[str, Any] | Request | None, None]:
        """Default callback for processing responses"""
        raise NotImplementedError(f"{self.__class__.__name__} must implement parse() method")
        yield  # Make this a generator for type checkers

    async def on_start(self, resuming: bool = False) -> None:
        """Called before crawling starts. Override for setup logic.

        :param resuming: It's enabled if the spider is resuming from a checkpoint, left for the user to use.
        """
        if resuming:
            self.logger.debug("Resuming spider from checkpoint")
        else:
            self.logger.debug("Starting spider")

    async def on_close(self) -> None:
        """Called after crawling finishes. Override for cleanup logic."""
        self.logger.debug("Spider closed")

    async def on_error(self, request: Request, error: Exception) -> None:
        """

View on GitHub (pinned to 5d213a2d47)

Solutions

Implement `async def parse(self, response)` as an async generator yielding Dict items, Requests, or None
Or always pass an explicit callback= when yielding Requests from a custom start_requests

Example fix

// before
class MySpider(Spider):
    name = "my"
    start_urls = ["https://example.com"]

// after
class MySpider(Spider):
    name = "my"
    start_urls = ["https://example.com"]

    async def parse(self, response):
        yield {"url": response.url, "status": response.status}

Defensive patterns

Strategy: validation

Validate before calling

assert MySpider.parse is not Spider.parse, "implement parse() or always pass callback="

Type guard

def implements_parse(spider_cls) -> bool:
    return spider_cls.parse is not Spider.parse

Prevention

Implement parse in every concrete spider
Pass an explicit callback on every yielded Request when parse is absent

When it happens

Trigger: Instantiating/crawling a Spider subclass that defines start_urls but no parse method; overriding start_requests to yield Request(url) without a callback, so the engine dispatches to the abstract parse.

Common situations: New spider classes where the author only wrote start_requests; renaming parse to something else (e.g. handle_response); subclassing an intermediate class that never implemented parse.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/745ac70b4f6b202b. Report an issue: GitHub.