{"record":{"id":"00c573ff4ce83273","repo":"D4Vinci/Scrapling","slug":"this-response-has-no-request-set-yet","errorCode":null,"errorMessage":"This response has no request set yet.","messagePattern":"This response has no request set yet\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"scrapling/engines/toolbelt/custom.py","lineNumber":118,"sourceCode":"\n        This is a helper method for spiders to easily follow links found in pages.\n\n        **IMPORTANT**: The below arguments if left empty, the corresponding value from the previous request will be used. The only exception is `dont_filter`.\n\n        :param url: The URL to follow (can be relative, will be joined with current URL)\n        :param sid: The session id to use\n        :param callback: Spider callback method to use\n        :param priority: The priority number to use, the higher the number, the higher priority to be processed first.\n        :param dont_filter: If this request has been done before, disable the filter to allow it again.\n        :param meta: Additional meta data to included in the request\n        :param referer_flow: Enabled by default, set the current response url as referer for the new request url.\n        :param kwargs: Additional Request arguments\n        :return: Request object ready to be yielded\n        \"\"\"\n        from scrapling.spiders import Request\n\n        if not self.request or not isinstance(self.request, Request):\n            raise TypeError(\"This response has no request set yet.\")\n\n        # Merge original session kwargs with new kwargs (new takes precedence)\n        session_kwargs = {**self.request._session_kwargs, **kwargs}\n\n        if referer_flow:\n            # For requests\n            headers = session_kwargs.get(\"headers\", {})\n            headers[\"referer\"] = self.url\n            session_kwargs[\"headers\"] = headers\n\n            # For browsers\n            extra_headers = session_kwargs.get(\"extra_headers\", {})\n            extra_headers[\"referer\"] = self.url\n            session_kwargs[\"extra_headers\"] = extra_headers\n\n            session_kwargs[\"google_search\"] = False\n\n        return Request(","sourceCodeStart":100,"sourceCodeEnd":136,"githubUrl":"https://github.com/D4Vinci/Scrapling/blob/5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f/scrapling/engines/toolbelt/custom.py#L100-L136","documentation":"Raised by Response.follow() when self.request is not set to a scrapling Request instance. follow() builds the next Request by merging the original request's stored _session_kwargs with new kwargs, so it can only work on responses produced by the crawler, which injects response.request after the fetch. Manually constructed Responses (or ones not yet processed by the crawler) have request=None and follow() rejects them with TypeError.","triggerScenarios":"Calling response.follow(url) on a Response you constructed yourself; calling follow() on a response obtained from a bare fetcher (Fetcher/StealthyFetcher .fetch/.get) outside a spider run; calling follow() before the crawler assigned response.request.","commonSituations":"Testing spider logic with hand-built Response fixtures; mixing the fetcher API with the crawler API and assuming follow() works everywhere; porting Scrapy habits (where response.follow also requires a live request) to standalone scrapling fetch calls.","solutions":["Inside a spider, only call follow() on responses passed to your callback by the crawler — they always have .request set.","Outside the crawler, construct the next Request yourself (from scrapling.spiders import Request) or just call the fetcher again with the new URL.","In tests, set response.request = Request(url, ...) before exercising follow()."],"exampleFix":"# before\nresp = fetcher.get('https://example.com')  # standalone fetch\nreq = resp.follow('/next')  # TypeError: no request set\n\n# after\nfrom scrapling.spiders import Request\nreq = Request('https://example.com/next', callback=self.parse_next)  # enqueue directly","handlingStrategy":"type-guard","validationCode":"from scrapling.spiders import Request\n\ndef can_follow(response) -> bool:\n    return isinstance(getattr(response, 'request', None), Request)","typeGuard":"from scrapling.spiders import Request\n\ndef can_follow(response) -> bool:\n    return isinstance(getattr(response, 'request', None), Request)","tryCatchPattern":"try:\n    req = response.follow(next_url)\nexcept TypeError as e:\n    if 'no request set' in str(e):\n        from scrapling.spiders import Request\n        req = Request(next_url, callback=self.parse)  # build it manually\n    else:\n        raise","preventionTips":["Only call follow() on crawler-delivered responses inside spider callbacks.","For standalone fetcher usage, enqueue a fresh Request instead.","In tests, assign response.request before exercising follow logic."],"tags":["response","crawler","spider","follow","validation"],"backgroundTag":null,"analyzedSha":"5d213a2d4764002bfc4fed33c32fe09fa8b0bf7f","analyzedAt":"2026-08-14T22:23:09.440Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}