{"record":{"id":"745ac70b4f6b202b","repo":"D4Vinci/Scrapling","slug":"self-class-name-must-implement-parse-m","errorCode":null,"errorMessage":"{self.__class__.__name__} must implement parse() method","messagePattern":"(.+?) must implement parse\\(\\) method","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"scrapling/spiders/spider.py","lineNumber":175,"sourceCode":"\n        By default, this generates Request objects for each URL in `start_urls`\n        using the session manager's default session and `parse()` as callback.\n\n        Override this method for more control over initial requests\n        (e.g., to add custom headers, use different callbacks, etc.)\n        \"\"\"\n        if not self.start_urls:\n            raise RuntimeError(\n                \"Spider has no starting point, either set `start_urls` or override `start_requests` function.\"\n            )\n\n        for url in self.start_urls:\n            yield Request(url, sid=self._session_manager.default_session_id)\n\n    @abstractmethod\n    async def parse(self, response: \"Response\") -> AsyncGenerator[Dict[str, Any] | Request | None, None]:\n        \"\"\"Default callback for processing responses\"\"\"\n        raise NotImplementedError(f\"{self.__class__.__name__} must implement parse() method\")\n        yield  # Make this a generator for type checkers\n\n    async def on_start(self, resuming: bool = False) -> None:\n        \"\"\"Called before crawling starts. Override for setup logic.\n\n        :param resuming: It's enabled if the spider is resuming from a checkpoint, left for the user to use.\n        \"\"\"\n        if resuming:\n            self.logger.debug(\"Resuming spider from checkpoint\")\n        else:\n            self.logger.debug(\"Starting spider\")\n\n    async def on_close(self) -> None:\n        \"\"\"Called after crawling finishes. Override for cleanup logic.\"\"\"\n        self.logger.debug(\"Spider closed\")\n\n    async def on_error(self, request: Request, error: Exception) -> None:\n        \"\"\"","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/spiders/spider.py#L157-L193","documentation":"Spider.parse is an abstract method; every concrete spider must implement it because it is the default callback for responses whose Request has no explicit callback. The base implementation raises NotImplementedError when called.","triggerScenarios":"Instantiating/crawling a Spider subclass that defines start_urls but no parse method; overriding start_requests to yield Request(url) without a callback, so the engine dispatches to the abstract parse.","commonSituations":"New spider classes where the author only wrote start_requests; renaming parse to something else (e.g. handle_response); subclassing an intermediate class that never implemented parse.","solutions":["Implement `async def parse(self, response)` as an async generator yielding Dict items, Requests, or None","Or always pass an explicit callback= when yielding Requests from a custom start_requests"],"exampleFix":"// before\nclass MySpider(Spider):\n    name = \"my\"\n    start_urls = [\"https://example.com\"]\n\n// after\nclass MySpider(Spider):\n    name = \"my\"\n    start_urls = [\"https://example.com\"]\n\n    async def parse(self, response):\n        yield {\"url\": response.url, \"status\": response.status}","handlingStrategy":"validation","validationCode":"assert MySpider.parse is not Spider.parse, \"implement parse() or always pass callback=\"","typeGuard":"def implements_parse(spider_cls) -> bool:\n    return spider_cls.parse is not Spider.parse","tryCatchPattern":null,"preventionTips":["Implement parse in every concrete spider","Pass an explicit callback on every yielded Request when parse is absent"],"tags":["spider","abstract-method","callback"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}