{"record":{"id":"f68a2d3f5caa438a","repo":"D4Vinci/Scrapling","slug":"sitemapspider-needs-sitemap-urls-to-be-set","errorCode":null,"errorMessage":"`SitemapSpider` needs `sitemap_urls` to be set.","messagePattern":"`SitemapSpider` needs `sitemap_urls` to be set\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"scrapling/spiders/templates/sitemap.py","lineNumber":71,"sourceCode":"    :cvar sitemap_alternate_links: When enabled, alternate-language URLs are also\n        routed through `rules()`.\n    \"\"\"\n\n    sitemap_urls: List[str] = []\n    sitemap_follow: Optional[LinkExtractor] = None\n    sitemap_alternate_links: bool = False\n\n    def rules(self) -> List[CrawlRule]:\n        \"\"\"Override to define dispatch rules for sitemap URLs.\"\"\"\n        return []\n\n    async def start_requests(self) -> AsyncGenerator[Request, None]:\n        if self.sitemap_urls:\n            for url in self.sitemap_urls:\n                yield Request(url, callback=self._parse_sitemap)\n            return\n\n        raise RuntimeError(\"`SitemapSpider` needs `sitemap_urls` to be set.\")\n\n    async def parse(self, response: \"Response\") -> AsyncGenerator[Union[Dict[str, Any], Request, None], None]:\n        \"\"\"Default callback for processing responses\"\"\"\n        raise NotImplementedError(f\"{self.__class__.__name__} must implement parse() method\")\n        yield  # Make this a generator for type checkers\n\n    def _robots_body(self, response: \"Response\") -> List[str]:\n        \"\"\"Extract `Sitemap` directives from a robots.txt body via protego.\"\"\"\n        try:\n            text = response.body.decode(response.encoding, errors=\"replace\")\n            parser = Protego.parse(text)\n        except Exception as e:\n            self.logger.warning(f\"Failed to parse robots.txt: {e}\")\n            return []\n        return list(parser.sitemaps)\n\n    def _extract_urls(self, root: Any) -> List[str]:\n        urls: List[str] = []","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/spiders/templates/sitemap.py#L53-L89","documentation":"SitemapSpider's default start_requests yields one Request per URL in the class attribute sitemap_urls, all pointed at _parse_sitemap. If sitemap_urls is empty/None, there is no starting point and it raises RuntimeError when the crawl begins.","triggerScenarios":"Subclassing SitemapSpider without setting sitemap_urls; setting it to an empty list; intending to discover sitemaps from robots.txt but not overriding start_requests to do so; class-attribute mutation clearing the list.","commonSituations":"Expecting automatic robots.txt discovery when the attribute is empty; typos in the attribute name; config-driven spiders where the URL list came back empty.","solutions":["Set sitemap_urls = [\"https://example.com/sitemap.xml\"] on the subclass","If you need dynamic discovery, override start_requests to yield the robots.txt URL with callback=self._parse_sitemap (the class ships a _robots_body helper that extracts Sitemap directives via protego)"],"exampleFix":"// before\nclass Site(SitemapSpider):\n    name = \"site\"\n    allowed_domains = [\"example.com\"]\n\n// after\nclass Site(SitemapSpider):\n    name = \"site\"\n    allowed_domains = [\"example.com\"]\n    sitemap_urls = [\"https://example.com/sitemap.xml\"]","handlingStrategy":"validation","validationCode":"if not getattr(Site, \"sitemap_urls\", None):\n    raise SystemExit(\"set sitemap_urls or override start_requests\")","typeGuard":"def has_sitemap_source(cls) -> bool:\n    return bool(getattr(cls, \"sitemap_urls\", None)) or cls.start_requests is not SitemapSpider.start_requests","tryCatchPattern":null,"preventionTips":["Set sitemap_urls on every sitemap spider","Override start_requests with the robots.txt URL for dynamic sitemap discovery"],"tags":["sitemap","spider-template","configuration","validation"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}