D4Vinci/Scrapling · error · RuntimeError

`SitemapSpider` needs `sitemap_urls` to be set.

Error message

`SitemapSpider` needs `sitemap_urls` to be set.

What it means

SitemapSpider's default start_requests yields one Request per URL in the class attribute sitemap_urls, all pointed at _parse_sitemap. If sitemap_urls is empty/None, there is no starting point and it raises RuntimeError when the crawl begins.

Source

Thrown at scrapling/spiders/templates/sitemap.py:71

    :cvar sitemap_alternate_links: When enabled, alternate-language URLs are also
        routed through `rules()`.
    """

    sitemap_urls: List[str] = []
    sitemap_follow: Optional[LinkExtractor] = None
    sitemap_alternate_links: bool = False

    def rules(self) -> List[CrawlRule]:
        """Override to define dispatch rules for sitemap URLs."""
        return []

    async def start_requests(self) -> AsyncGenerator[Request, None]:
        if self.sitemap_urls:
            for url in self.sitemap_urls:
                yield Request(url, callback=self._parse_sitemap)
            return

        raise RuntimeError("`SitemapSpider` needs `sitemap_urls` to be set.")

    async def parse(self, response: "Response") -> AsyncGenerator[Union[Dict[str, Any], Request, None], None]:
        """Default callback for processing responses"""
        raise NotImplementedError(f"{self.__class__.__name__} must implement parse() method")
        yield  # Make this a generator for type checkers

    def _robots_body(self, response: "Response") -> List[str]:
        """Extract `Sitemap` directives from a robots.txt body via protego."""
        try:
            text = response.body.decode(response.encoding, errors="replace")
            parser = Protego.parse(text)
        except Exception as e:
            self.logger.warning(f"Failed to parse robots.txt: {e}")
            return []
        return list(parser.sitemaps)

    def _extract_urls(self, root: Any) -> List[str]:
        urls: List[str] = []

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Set sitemap_urls = ["https://example.com/sitemap.xml"] on the subclass
2. If you need dynamic discovery, override start_requests to yield the robots.txt URL with callback=self._parse_sitemap (the class ships a _robots_body helper that extracts Sitemap directives via protego)

Example fix

// before
class Site(SitemapSpider):
    name = "site"
    allowed_domains = ["example.com"]

// after
class Site(SitemapSpider):
    name = "site"
    allowed_domains = ["example.com"]
    sitemap_urls = ["https://example.com/sitemap.xml"]

if not getattr(Site, "sitemap_urls", None):
    raise SystemExit("set sitemap_urls or override start_requests")

def has_sitemap_source(cls) -> bool:
    return bool(getattr(cls, "sitemap_urls", None)) or cls.start_requests is not SitemapSpider.start_requests

When it happens

Trigger: Subclassing SitemapSpider without setting sitemap_urls; setting it to an empty list; intending to discover sitemaps from robots.txt but not overriding start_requests to do so; class-attribute mutation clearing the list.

Common situations: Expecting automatic robots.txt discovery when the attribute is empty; typos in the attribute name; config-driven spiders where the URL list came back empty.

Related errors

• {self.__class__.__name__} must set `target_website`, `start_
• Cannot use 'proxy_rotator' together with 'proxy' or 'proxies
• Unknown parser argument: "{key}"; maybe you meant {cls.parse
• You must pass a keyword to configure, current keywords: {cls
• Invalid proxy string!

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/f68a2d3f5caa438a. Report an issue: GitHub.