D4Vinci/Scrapling · error · ValueError

{self.__class__.__name__} must set `target_website`, `start_

Error message

{self.__class__.__name__} must set `target_website`, `start_urls`, or `allowed_domains`

What it means

The Shopify template derives the target store domain from target_website, start_urls, or allowed_domains (first non-empty value, in that order). If all three are unset/empty, __init__ raises ValueError because there is no way to build the Shopify collections/products URLs.

Source

Thrown at scrapling/spiders/templates/shopify.py:43

class ShopifySpider(Spider):
    """A spider that extracts all products from any Shopify-powered website through its JSON API.

    Set `target_website` to the store's domain (or set `start_urls`/`allowed_domains` instead), and the
    spider walks the store's `/collections.json` pages, then each collection's `products.json` pages,
    yielding one item per product variant without touching the website's HTML.
    """

    name = "shopify"
    target_website = ""
    collections_url = "https://{website}/collections.json?page={page}&limit=250"
    products_url = "https://{website}/collections/{handle}/products.json?page={page}&limit=250"
    product_url = "https://{website}/collections/{handle}/products/{product_handle}"

    def __init__(self, *args: Any, **kwargs: Any):
        super().__init__(*args, **kwargs)
        source = self.target_website or next(iter(self.start_urls or ()), "") or next(iter(self.allowed_domains), "")
        if not source:
            raise ValueError(f"{self.__class__.__name__} must set `target_website`, `start_urls`, or `allowed_domains`")
        self.target_website = urlparse(source if "://" in source else f"https://{source}").netloc
        self.collected_ids: Set[int] = set()

    async def start_requests(self) -> AsyncGenerator[Request, None]:
        yield Request(
            self.collections_url.format(website=self.target_website, page=1),
            callback=self.parse,
            meta={"page": 1},
        )

    async def parse(self, response: "Response") -> AsyncGenerator[Union[Dict[str, Any], Request, None], None]:
        collections = response.json()["collections"]
        if collections:
            for collection in collections:
                if collection["products_count"]:
                    yield Request(
                        self.products_url.format(website=self.target_website, handle=collection["handle"], page=1),
                        callback=self.parse_collection,

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Set target_website = "store.myshopify.com" on the subclass (bare host or full URL both work; netloc is extracted)
2. Alternatively provide start_urls or allowed_domains containing the store domain

Example fix

// before
class Store(ShopifySpider):
    name = "store"

// after
class Store(ShopifySpider):
    name = "store"
    target_website = "store.myshopify.com"

if not (target_website or start_urls or allowed_domains):
    raise ValueError("set target_website, start_urls, or allowed_domains")

When it happens

Trigger: Subclassing the Shopify spider without setting target_website, start_urls, or allowed_domains; setting start_urls = [] and allowed_domains = [] explicitly; setting attributes with different names (e.g. domain = "store.myshopify.com").

Common situations: Quick scaffolds of the template; renaming fields; expecting the base Spider defaults to carry over.

Related errors

• `SitemapSpider` needs `sitemap_urls` to be set.
• Cannot use 'proxy_rotator' together with 'proxy' or 'proxies
• Unknown parser argument: "{key}"; maybe you meant {cls.parse
• You must pass a keyword to configure, current keywords: {cls
• Invalid proxy string!

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/94fee4e35cd2968c. Report an issue: GitHub.