D4Vinci/Scrapling · error · TypeError

This response has no request set yet.

Error message

This response has no request set yet.

What it means

Raised by Response.follow() when self.request is not set to a scrapling Request instance. follow() builds the next Request by merging the original request's stored _session_kwargs with new kwargs, so it can only work on responses produced by the crawler, which injects response.request after the fetch. Manually constructed Responses (or ones not yet processed by the crawler) have request=None and follow() rejects them with TypeError.

Source

Thrown at scrapling/engines/toolbelt/custom.py:118

        This is a helper method for spiders to easily follow links found in pages.

        **IMPORTANT**: The below arguments if left empty, the corresponding value from the previous request will be used. The only exception is `dont_filter`.

        :param url: The URL to follow (can be relative, will be joined with current URL)
        :param sid: The session id to use
        :param callback: Spider callback method to use
        :param priority: The priority number to use, the higher the number, the higher priority to be processed first.
        :param dont_filter: If this request has been done before, disable the filter to allow it again.
        :param meta: Additional meta data to included in the request
        :param referer_flow: Enabled by default, set the current response url as referer for the new request url.
        :param kwargs: Additional Request arguments
        :return: Request object ready to be yielded
        """
        from scrapling.spiders import Request

        if not self.request or not isinstance(self.request, Request):
            raise TypeError("This response has no request set yet.")

        # Merge original session kwargs with new kwargs (new takes precedence)
        session_kwargs = {**self.request._session_kwargs, **kwargs}

        if referer_flow:
            # For requests
            headers = session_kwargs.get("headers", {})
            headers["referer"] = self.url
            session_kwargs["headers"] = headers

            # For browsers
            extra_headers = session_kwargs.get("extra_headers", {})
            extra_headers["referer"] = self.url
            session_kwargs["extra_headers"] = extra_headers

            session_kwargs["google_search"] = False

        return Request(

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Inside a spider, only call follow() on responses passed to your callback by the crawler — they always have .request set.
2. Outside the crawler, construct the next Request yourself (from scrapling.spiders import Request) or just call the fetcher again with the new URL.
3. In tests, set response.request = Request(url, ...) before exercising follow().

Example fix

# before
resp = fetcher.get('https://example.com')  # standalone fetch
req = resp.follow('/next')  # TypeError: no request set

# after
from scrapling.spiders import Request
req = Request('https://example.com/next', callback=self.parse_next)  # enqueue directly

from scrapling.spiders import Request

def can_follow(response) -> bool:
    return isinstance(getattr(response, 'request', None), Request)

from scrapling.spiders import Request

def can_follow(response) -> bool:
    return isinstance(getattr(response, 'request', None), Request)

try:
    req = response.follow(next_url)
except TypeError as e:
    if 'no request set' in str(e):
        from scrapling.spiders import Request
        req = Request(next_url, callback=self.parse)  # build it manually
    else:
        raise

When it happens

Trigger: Calling response.follow(url) on a Response you constructed yourself; calling follow() on a response obtained from a bare fetcher (Fetcher/StealthyFetcher .fetch/.get) outside a spider run; calling follow() before the crawler assigned response.request.

Common situations: Testing spider logic with hand-built Response fixtures; mixing the fetcher API with the crawler API and assuming follow() works everywhere; porting Scrapy habits (where response.follow also requires a live request) to standalone scrapling fetch calls.

Related errors

• Response meta should be dictionary but got {type(meta).__nam
• {self.__class__.__name__} must have a name.
• Spider has no starting point, either set `start_urls` or ove
• 'quality' is only valid when 'image_type' is 'jpeg'.
• Unknown extraction type: {extraction_type}

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/00c573ff4ce83273. Report an issue: GitHub.