What causes "{self.__class__.__name__} must implement parse() method" in D4Vinci/Scrapling?

Implement `async def parse(self, response)` as an async generator yielding Dict items, Requests, or None

D4Vinci/Scrapling · error · NotImplementedError

must implement parse() method

Error message

{self.__class__.__name__} must implement parse() method

What it means

Spider.parse is an abstract method; every concrete spider must implement it because it is the default callback for responses whose Request has no explicit callback. The base implementation raises NotImplementedError when called.

Solutions

Implement `async def parse(self, response)` as an async generator yielding Dict items, Requests, or None
Or always pass an explicit callback= when yielding Requests from a custom start_requests

Example fix

// before
class MySpider(Spider):
    name = "my"
    start_urls = ["https://example.com"]

// after
class MySpider(Spider):
    name = "my"
    start_urls = ["https://example.com"]

    async def parse(self, response):
        yield {"url": response.url, "status": response.status}

Defensive patterns

Strategy: validation

Validate before calling

assert MySpider.parse is not Spider.parse, "implement parse() or always pass callback="

Type guard

def implements_parse(spider_cls) -> bool:
    return spider_cls.parse is not Spider.parse

Prevention

Implement parse in every concrete spider
Pass an explicit callback on every yielded Request when parse is absent

When it happens

Trigger: Instantiating/crawling a Spider subclass that defines start_urls but no parse method; overriding start_requests to yield Request(url) without a callback, so the engine dispatches to the abstract parse.

Common situations: New spider classes where the author only wrote start_requests; renaming parse to something else (e.g. handle_response); subclassing an intermediate class that never implemented parse.

Related errors

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/745ac70b4f6b202b. Report an issue: GitHub.

Appendix: source

Thrown at scrapling/spiders/spider.py:175

        By default, this generates Request objects for each URL in `start_urls`
        using the session manager's default session and `parse()` as callback.

        Override this method for more control over initial requests
        (e.g., to add custom headers, use different callbacks, etc.)
        """
        if not self.start_urls:
            raise RuntimeError(
                "Spider has no starting point, either set `start_urls` or override `start_requests` function."
            )

        for url in self.start_urls:
            yield Request(url, sid=self._session_manager.default_session_id)

    @abstractmethod
    async def parse(self, response: "Response") -> AsyncGenerator[Dict[str, Any] | Request | None, None]:
        """Default callback for processing responses"""
        raise NotImplementedError(f"{self.__class__.__name__} must implement parse() method")
        yield  # Make this a generator for type checkers

    async def on_start(self, resuming: bool = False) -> None:
        """Called before crawling starts. Override for setup logic.

        :param resuming: It's enabled if the spider is resuming from a checkpoint, left for the user to use.
        """
        if resuming:
            self.logger.debug("Resuming spider from checkpoint")
        else:
            self.logger.debug("Starting spider")

    async def on_close(self) -> None:
        """Called after crawling finishes. Override for cleanup logic."""
        self.logger.debug("Spider closed")

    async def on_error(self, request: Request, error: Exception) -> None:
        """

View on GitHub (pinned to 5d213a2d47)