D4Vinci/Scrapling · error · NotImplementedError

{self.__class__.__name__} must implement parse_node() method

Error message

{self.__class__.__name__} must implement parse_node() method

What it means

XMLFeedSpider-derived spiders must override parse_node(); the base implementation raises NotImplementedError. The engine calls parse_node(response, node) once per matched itertag element found in the feed.

Source

Thrown at scrapling/spiders/templates/feed.py:71

        except OSError as e:
            self.logger.warning(f"Failed to decompress feed: {e}")
            return

        try:
            root = etree.fromstring(body)
        except etree.XMLSyntaxError as e:
            self.logger.warning(f"Failed to parse XML feed from {response.url}: {e}")
            return

        for node in self._iter_nodes(root):
            async for result in self.parse_node(response, node):
                yield result

    async def parse_node(
        self, response: "Response", node: _Element
    ) -> AsyncGenerator[Union[Dict[str, Any], Request, None], None]:
        """Override to process one feed node; `node` is a namespace-stripped `lxml` element."""
        raise NotImplementedError(f"{self.__class__.__name__} must implement parse_node() method")
        yield  # Make this a generator for type checkers

    def _wanted_tag(self) -> Tuple[Optional[str], str]:
        """Resolve `itertag` into a `(namespace uri or None, localname)` pair."""
        prefix, _, name = self.itertag.rpartition(":")
        if not prefix:
            return None, name
        uri = dict(self.namespaces).get(prefix)
        if not uri:
            raise ValueError(f"`itertag` prefix {prefix!r} is not defined in `namespaces`")
        return uri, name

    def _iter_nodes(self, root: _Element) -> Iterator[_Element]:
        uri, name = self._wanted_tag()
        for el in root.iter():
            if isinstance(el.tag, str):
                qname = etree.QName(el.tag)
                if qname.localname == name and (uri is None or qname.namespace == uri):

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Implement `async def parse_node(self, response, node)` yielding items/Requests/None, where node is a namespace-stripped lxml element
2. Verify the exact method name and signature against the template docs before running

Example fix

// before
class Feed(XMLFeedSpider):
    name = "feed"
    itertag = "item"

// after
class Feed(XMLFeedSpider):
    name = "feed"
    itertag = "item"

    async def parse_node(self, response, node):
        yield {"id": node.get("id")}

assert Feed.parse_node is not XMLFeedSpider.parse_node, "implement parse_node()"

def implements_parse_node(cls) -> bool:
    return cls.parse_node is not XMLFeedSpider.parse_node

When it happens

Trigger: Subclassing the XML feed template, setting itertag/namespaces, but not implementing parse_node; renaming the method (e.g. parse_item) so the abstract base is what gets dispatched.

Common situations: Porting a Scrapy XMLFeedSpider and assuming a different hook name; scaffolding the template subclass without filling in the node handler.

Related errors

• {self.__class__.__name__} must implement parse_row() method
• `itertag` prefix {prefix!r} is not defined in `namespaces`
• {self.__class__.__name__} must implement parse() method
• Storage system must implement `save` method
• Storage system must implement `retrieve` method

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/5d81599624d93c6f. Report an issue: GitHub.