{"record":{"id":"53d7573a30616531","repo":"scrapy/scrapy","slug":"encoding-can-t-be-none","errorCode":null,"errorMessage":"encoding can't be None","messagePattern":"encoding can't be None","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scrapy/http/response/__init__.py","lineNumber":257,"sourceCode":"        encoding: str | None = \"utf-8\",\n        priority: int = 0,\n        dont_filter: bool = False,\n        errback: Callable[[Failure], Any] | None = None,\n        cb_kwargs: dict[str, Any] | None = None,\n        flags: list[str] | None = None,\n    ) -> Request:\n        \"\"\"\n        Return a :class:`~.Request` instance to follow a link ``url``.\n        It accepts the same arguments as ``Request.__init__()`` method,\n        but ``url`` can be a relative URL or a :class:`~scrapy.link.Link` object,\n        not only an absolute URL.\n\n        :class:`~.TextResponse` provides a :meth:`~.TextResponse.follow`\n        method which supports selectors in addition to absolute/relative URLs\n        and Link objects.\n        \"\"\"\n        if encoding is None:\n            raise ValueError(\"encoding can't be None\")\n        if isinstance(url, Link):\n            url = url.url\n        elif url is None:\n            raise ValueError(\"url can't be None\")\n        url = self.urljoin(url)\n\n        return Request(\n            url=url,\n            callback=callback,\n            method=method,\n            headers=headers,\n            body=body,\n            cookies=cookies,\n            meta=meta,\n            encoding=encoding,\n            priority=priority,\n            dont_filter=dont_filter,\n            errback=errback,","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/scrapy/scrapy/blob/06af687662112027b4482d31e2714a3cf280a91f/scrapy/http/response/__init__.py#L239-L275","documentation":"Response.follow() requires an explicit encoding argument (unlike TextResponse.follow, which defaults it to the response encoding). The base implementation raises ValueError if encoding is None because it must build the Request body/URL handling deterministically without guessing a charset.","triggerScenarios":"Calling response.follow('/next') on a base Response without passing encoding='utf-8'; code that was written against TextResponse but now receives plain Response; generic helpers calling follow() with default encoding.","commonSituations":"Callbacks handling both text and binary responses calling follow uniformly; refactors that changed response classes; tests using bare Response.","solutions":["Pass an explicit encoding: response.follow('/next', callback=..., encoding='utf-8').","Use a TextResponse subclass so encoding defaults to the response's detected encoding.","In shared helpers branch on isinstance(response, TextResponse).","Read encoding from response headers when available and forward it."],"exampleFix":"# before\nyield response.follow(\"/next\", callback=self.parse)  # ValueError on Response\n\n# after\nyield response.follow(\"/next\", callback=self.parse, encoding=\"utf-8\")\n","handlingStrategy":"validation","validationCode":"encoding = encoding or getattr(response, \"encoding\", None) or \"utf-8\"\nyield response.follow(url, callback=self.parse, encoding=encoding)","typeGuard":"from scrapy.http import Response, TextResponse\n\ndef can_follow_without_encoding(resp: Response) -> bool:\n    return isinstance(resp, TextResponse)","tryCatchPattern":"try:\n    yield response.follow(href, callback=self.parse)\nexcept ValueError:\n    yield response.follow(href, callback=self.parse, encoding=\"utf-8\")","preventionTips":["Always pass encoding explicitly when calling follow() on non-text responses.","Centralize link-following in one helper that supplies a default encoding.","Prefer TextResponse.follow when the response class is textual."],"tags":["scrapy","response","follow","encoding","valueerror"],"backgroundTag":null,"analyzedSha":"06af687662112027b4482d31e2714a3cf280a91f","analyzedAt":"2026-08-15T00:21:48.041Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}