What causes "encoding can't be None" in scrapy/scrapy?

Pass an explicit encoding: response.follow('/next', callback=..., encoding='utf-8').

scrapy/scrapy · error · ValueError

encoding can't be None

Error message

encoding can't be None

What it means

Response.follow() requires an explicit encoding argument (unlike TextResponse.follow, which defaults it to the response encoding). The base implementation raises ValueError if encoding is None because it must build the Request body/URL handling deterministically without guessing a charset.

Source

Thrown at scrapy/http/response/__init__.py:257

        encoding: str | None = "utf-8",
        priority: int = 0,
        dont_filter: bool = False,
        errback: Callable[[Failure], Any] | None = None,
        cb_kwargs: dict[str, Any] | None = None,
        flags: list[str] | None = None,
    ) -> Request:
        """
        Return a :class:`~.Request` instance to follow a link ``url``.
        It accepts the same arguments as ``Request.__init__()`` method,
        but ``url`` can be a relative URL or a :class:`~scrapy.link.Link` object,
        not only an absolute URL.

        :class:`~.TextResponse` provides a :meth:`~.TextResponse.follow`
        method which supports selectors in addition to absolute/relative URLs
        and Link objects.
        """
        if encoding is None:
            raise ValueError("encoding can't be None")
        if isinstance(url, Link):
            url = url.url
        elif url is None:
            raise ValueError("url can't be None")
        url = self.urljoin(url)

        return Request(
            url=url,
            callback=callback,
            method=method,
            headers=headers,
            body=body,
            cookies=cookies,
            meta=meta,
            encoding=encoding,
            priority=priority,
            dont_filter=dont_filter,
            errback=errback,

View on GitHub (pinned to 06af687662)

Solutions

Pass an explicit encoding: response.follow('/next', callback=..., encoding='utf-8').
Use a TextResponse subclass so encoding defaults to the response's detected encoding.
In shared helpers branch on isinstance(response, TextResponse).
Read encoding from response headers when available and forward it.

Example fix

# before
yield response.follow("/next", callback=self.parse)  # ValueError on Response

# after
yield response.follow("/next", callback=self.parse, encoding="utf-8")

Defensive patterns

Strategy: validation

Validate before calling

encoding = encoding or getattr(response, "encoding", None) or "utf-8"
yield response.follow(url, callback=self.parse, encoding=encoding)

Type guard

from scrapy.http import Response, TextResponse

def can_follow_without_encoding(resp: Response) -> bool:
    return isinstance(resp, TextResponse)

Try / catch

try:
    yield response.follow(href, callback=self.parse)
except ValueError:
    yield response.follow(href, callback=self.parse, encoding="utf-8")

Prevention

Always pass encoding explicitly when calling follow() on non-text responses.
Centralize link-following in one helper that supplies a default encoding.
Prefer TextResponse.follow when the response class is textual.

When it happens

Trigger: Calling response.follow('/next') on a base Response without passing encoding='utf-8'; code that was written against TextResponse but now receives plain Response; generic helpers calling follow() with default encoding.

Common situations: Callbacks handling both text and binary responses calling follow uniformly; refactors that changed response classes; tests using bare Response.

Related errors

AI-assisted analysis of scrapy/scrapy@06af687662 (2026-08-15). Data as JSON: /api/errors/53d7573a30616531. Report an issue: GitHub.