D4Vinci/Scrapling · error · NotImplementedError

{self.__class__.__name__} must implement parse() method

Error message

{self.__class__.__name__} must implement parse() method

What it means

SitemapSpider's default parse callback raises NotImplementedError — it exists only to satisfy the Spider contract. Sitemap pages are dispatched via rules()/callbacks; responses that fall through to the default parse hit this error.

Source

Thrown at scrapling/spiders/templates/sitemap.py:75

    sitemap_urls: List[str] = []
    sitemap_follow: Optional[LinkExtractor] = None
    sitemap_alternate_links: bool = False

    def rules(self) -> List[CrawlRule]:
        """Override to define dispatch rules for sitemap URLs."""
        return []

    async def start_requests(self) -> AsyncGenerator[Request, None]:
        if self.sitemap_urls:
            for url in self.sitemap_urls:
                yield Request(url, callback=self._parse_sitemap)
            return

        raise RuntimeError("`SitemapSpider` needs `sitemap_urls` to be set.")

    async def parse(self, response: "Response") -> AsyncGenerator[Union[Dict[str, Any], Request, None], None]:
        """Default callback for processing responses"""
        raise NotImplementedError(f"{self.__class__.__name__} must implement parse() method")
        yield  # Make this a generator for type checkers

    def _robots_body(self, response: "Response") -> List[str]:
        """Extract `Sitemap` directives from a robots.txt body via protego."""
        try:
            text = response.body.decode(response.encoding, errors="replace")
            parser = Protego.parse(text)
        except Exception as e:
            self.logger.warning(f"Failed to parse robots.txt: {e}")
            return []
        return list(parser.sitemaps)

    def _extract_urls(self, root: Any) -> List[str]:
        urls: List[str] = []
        for url_el in root:
            if self._get_type(url_el) != "url":
                continue

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Define the callbacks you actually use and implement them (e.g. parse_page) — do not rely on the base parse
2. Review rules() so every followed URL type has a matching rule with a callback; always pass callback= when yielding new Requests
3. If a default handler is genuinely needed, override parse() in the subclass

Example fix

// before
class Site(SitemapSpider):
    name = "site"
    sitemap_urls = ["https://example.com/sitemap.xml"]
    def rules(self):
        return [Rule("example.com/page/", callback="parse_page")]

// after
class Site(SitemapSpider):
    name = "site"
    sitemap_urls = ["https://example.com/sitemap.xml"]
    def rules(self):
        return [Rule("example.com/page/", callback="parse_page")]

    async def parse_page(self, response):
        yield {"url": response.url}

# every yielded Request must carry a callback on sitemap spiders
yield Request(next_url, callback=self.parse_page)

def has_default_parse(cls) -> bool:
    return cls.parse is not SitemapSpider.parse

When it happens

Trigger: A crawled URL reached the sitemap spider with no matching rule in rules() and no explicit callback, so the engine falls back to parse(); subclassing SitemapSpider expecting to implement parse() like a plain Spider but never overriding it.

Common situations: Rules whose patterns don't match followed URLs (typo, wrong regex, missing domain); Requests yielded from callbacks without callback= specified; porting a plain Spider to the sitemap template and keeping parse.

Related errors

• {self.__class__.__name__} must implement parse() method
• {self.__class__.__name__} must implement parse_node() method
• {self.__class__.__name__} must implement parse_row() method
• `SitemapSpider` needs `sitemap_urls` to be set.
• Storage system must implement `save` method

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/8bb6640b0bb3774b. Report an issue: GitHub.