D4Vinci/Scrapling · error · RuntimeError

Spider has no starting point, either set `start_urls` or ove

Error message

Spider has no starting point, either set `start_urls` or override `start_requests` function.

What it means

The default Spider.start_requests() requires at least one entry in the class attribute start_urls. If it's empty/None and start_requests isn't overridden, there is nothing to crawl and the engine raises at crawl start.

Source

Thrown at scrapling/spiders/spider.py:165

        except Exception as e:
            raise SessionConfigurationError(f"Error in {self.__class__.__name__}.configure_sessions(): {e}") from e

        if len(self._session_manager) == 0:
            raise SessionConfigurationError(f"{self.__class__.__name__}.configure_sessions() did not add any sessions")

        self.logger.info("Spider initialized")

    async def start_requests(self) -> AsyncGenerator[Request, None]:
        """Generate initial requests to start the crawl.

        By default, this generates Request objects for each URL in `start_urls`
        using the session manager's default session and `parse()` as callback.

        Override this method for more control over initial requests
        (e.g., to add custom headers, use different callbacks, etc.)
        """
        if not self.start_urls:
            raise RuntimeError(
                "Spider has no starting point, either set `start_urls` or override `start_requests` function."
            )

        for url in self.start_urls:
            yield Request(url, sid=self._session_manager.default_session_id)

    @abstractmethod
    async def parse(self, response: "Response") -> AsyncGenerator[Dict[str, Any] | Request | None, None]:
        """Default callback for processing responses"""
        raise NotImplementedError(f"{self.__class__.__name__} must implement parse() method")
        yield  # Make this a generator for type checkers

    async def on_start(self, resuming: bool = False) -> None:
        """Called before crawling starts. Override for setup logic.

        :param resuming: It's enabled if the spider is resuming from a checkpoint, left for the user to use.
        """
        if resuming:

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Set start_urls = ["https://example.com"] as a class attribute
2. Or override `async def start_requests(self)` and yield Request(url, sid=self._session_manager.default_session_id) yourself
3. Verify the attribute is exactly `start_urls` (plural) and non-empty at runtime

Example fix

// before
class MySpider(Spider):
    name = "my"
    async def parse(self, response): ...

// after
class MySpider(Spider):
    name = "my"
    start_urls = ["https://example.com"]
    async def parse(self, response): ...

if not getattr(MySpider, "start_urls", None) and Spider.start_requests is MySpider.start_requests:
    raise SystemExit("set start_urls or override start_requests")

def has_start_point(spider_cls) -> bool:
    return bool(getattr(spider_cls, "start_urls", None)) or spider_cls.start_requests is not Spider.start_requests

When it happens

Trigger: Defining a spider with neither start_urls nor a start_requests override; setting start_urls = [] (empty list) expecting requests to come from elsewhere; overriding __init__ and overwriting start_urls with an empty value.

Common situations: Spiders that intend to generate requests dynamically but forgot to override start_requests; config-driven spiders where the URL list came back empty; renaming the attribute (e.g. start_url singular).

Related errors

• {self.__class__.__name__} must have a name.
• Cannot use 'proxy_rotator' together with 'proxy' or 'proxies
• This response has no request set yet.
• Unknown parser argument: "{key}"; maybe you meant {cls.parse
• You must pass a keyword to configure, current keywords: {cls

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/72c216237df95a1a. Report an issue: GitHub.