{"record":{"id":"8bb6640b0bb3774b","repo":"D4Vinci/Scrapling","slug":"self-class-name-must-implement-parse-m-8bb664","errorCode":null,"errorMessage":"{self.__class__.__name__} must implement parse() method","messagePattern":"(.+?) must implement parse\\(\\) method","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"scrapling/spiders/templates/sitemap.py","lineNumber":75,"sourceCode":"    sitemap_urls: List[str] = []\n    sitemap_follow: Optional[LinkExtractor] = None\n    sitemap_alternate_links: bool = False\n\n    def rules(self) -> List[CrawlRule]:\n        \"\"\"Override to define dispatch rules for sitemap URLs.\"\"\"\n        return []\n\n    async def start_requests(self) -> AsyncGenerator[Request, None]:\n        if self.sitemap_urls:\n            for url in self.sitemap_urls:\n                yield Request(url, callback=self._parse_sitemap)\n            return\n\n        raise RuntimeError(\"`SitemapSpider` needs `sitemap_urls` to be set.\")\n\n    async def parse(self, response: \"Response\") -> AsyncGenerator[Union[Dict[str, Any], Request, None], None]:\n        \"\"\"Default callback for processing responses\"\"\"\n        raise NotImplementedError(f\"{self.__class__.__name__} must implement parse() method\")\n        yield  # Make this a generator for type checkers\n\n    def _robots_body(self, response: \"Response\") -> List[str]:\n        \"\"\"Extract `Sitemap` directives from a robots.txt body via protego.\"\"\"\n        try:\n            text = response.body.decode(response.encoding, errors=\"replace\")\n            parser = Protego.parse(text)\n        except Exception as e:\n            self.logger.warning(f\"Failed to parse robots.txt: {e}\")\n            return []\n        return list(parser.sitemaps)\n\n    def _extract_urls(self, root: Any) -> List[str]:\n        urls: List[str] = []\n        for url_el in root:\n            if self._get_type(url_el) != \"url\":\n                continue\n","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/spiders/templates/sitemap.py#L57-L93","documentation":"SitemapSpider's default parse callback raises NotImplementedError — it exists only to satisfy the Spider contract. Sitemap pages are dispatched via rules()/callbacks; responses that fall through to the default parse hit this error.","triggerScenarios":"A crawled URL reached the sitemap spider with no matching rule in rules() and no explicit callback, so the engine falls back to parse(); subclassing SitemapSpider expecting to implement parse() like a plain Spider but never overriding it.","commonSituations":"Rules whose patterns don't match followed URLs (typo, wrong regex, missing domain); Requests yielded from callbacks without callback= specified; porting a plain Spider to the sitemap template and keeping parse.","solutions":["Define the callbacks you actually use and implement them (e.g. parse_page) — do not rely on the base parse","Review rules() so every followed URL type has a matching rule with a callback; always pass callback= when yielding new Requests","If a default handler is genuinely needed, override parse() in the subclass"],"exampleFix":"// before\nclass Site(SitemapSpider):\n    name = \"site\"\n    sitemap_urls = [\"https://example.com/sitemap.xml\"]\n    def rules(self):\n        return [Rule(\"example.com/page/\", callback=\"parse_page\")]\n\n// after\nclass Site(SitemapSpider):\n    name = \"site\"\n    sitemap_urls = [\"https://example.com/sitemap.xml\"]\n    def rules(self):\n        return [Rule(\"example.com/page/\", callback=\"parse_page\")]\n\n    async def parse_page(self, response):\n        yield {\"url\": response.url}","handlingStrategy":"validation","validationCode":"# every yielded Request must carry a callback on sitemap spiders\nyield Request(next_url, callback=self.parse_page)","typeGuard":"def has_default_parse(cls) -> bool:\n    return cls.parse is not SitemapSpider.parse","tryCatchPattern":null,"preventionTips":["Give every rule a callback and every yielded Request an explicit callback","Verify rule patterns match the URLs actually followed","Override parse only if a true default handler is needed"],"tags":["sitemap","abstract-method","callback","spider-template"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}