scrapy/scrapy · error · NotConfigured

{type(self).name} requires the asyncio support. Make sur

Error message

{type(self).__name__} requires the asyncio support. Make sure that you have either enabled the asyncio Twisted reactor in the TWISTED_REACTOR setting or disabled the TWISTED_REACTOR_ENABLED setting. See the asyncio documentation of Scrapy for more information.

What it means

Streaming download handlers (the new async streaming HTTP handlers, e.g. the experimental ones) require Scrapy running on the asyncio Twisted reactor. Their __init__ checks is_asyncio_available() — true only when TWISTED_REACTOR selects the asyncio reactor or TWISTED_REACTOR_ENABLED is false — and raises NotConfigured with this message otherwise. Because it is NotConfigured, the handler is skipped at startup and the failure surfaces later as an unsupported-scheme download error.

Source

Thrown at scrapy/core/downloader/handlers/_base_streaming.py:70

    headers: Headers
    certificate: NotRequired[Any]
    ip_address: NotRequired[IPv4Address | IPv6Address | None]
    protocol: str | None


class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_ResponseT]):
    """A base class for HTTP download handlers that follow the streaming logic flow."""

    _DEFAULT_CONNECT_TIMEOUT: ClassVar[float] = 10
    experimental: ClassVar[bool] = False
    requires_asyncio: ClassVar[bool] = True
    # require subclasses to disable proxies explicitly with an explanation
    supports_proxies: ClassVar[bool] = True
    supports_per_request_bindaddress: ClassVar[bool] = False

    def __init__(self, crawler: Crawler):
        if self.requires_asyncio and not is_asyncio_available():  # pragma: no cover
            raise NotConfigured(
                f"{type(self).__name__} requires the asyncio support. Make"
                f" sure that you have either enabled the asyncio Twisted"
                f" reactor in the TWISTED_REACTOR setting or disabled the"
                f" TWISTED_REACTOR_ENABLED setting. See the asyncio documentation"
                f" of Scrapy for more information."
            )
        self._check_deps_installed()
        super().__init__(crawler)
        if self.experimental:
            logger.warning(
                f"{type(self).__name__} is experimental and is not recommended for production use."
            )
        self._bind_address = normalize_bind_address(
            crawler.settings.get("DOWNLOAD_BIND_ADDRESS")
        )
        self._proxy_auth_encoding: str = crawler.settings.get("HTTPPROXY_AUTH_ENCODING")
        # these are useful for many handlers but used in different ways by them
        self._pool_size_total: int = crawler.settings.getint("CONCURRENT_REQUESTS")

View on GitHub (pinned to 06af687662)

Solutions

Enable the asyncio reactor: TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor' in settings.
Or set TWISTED_REACTOR_ENABLED = False to let Scrapy manage reactor selection (asyncio used by default in recent versions).
If you must stay on the default reactor, revert the DOWNLOAD_HANDLERS override to the classic handler for that scheme.

Example fix

# before
DOWNLOAD_HANDLERS = {
    'https': 'myproj.handlers.MyStreamingHandler',
}
# TWISTED_REACTOR unset -> NotConfigured: requires the asyncio support

# after
TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'
DOWNLOAD_HANDLERS = {
    'https': 'myproj.handlers.MyStreamingHandler',
}

Defensive patterns

Strategy: validation

Validate before calling

from scrapy.utils.reactor import is_asyncio_available
import scrapy
from scrapy.utils.test import get_crawler

assert is_asyncio_available(), (
    "set TWISTED_REACTOR='twisted.internet.asyncioreactor.AsyncioSelectorReactor' "
    "before using streaming handlers"
)

Prevention

Set the asyncio reactor in settings whenever you enable streaming handlers.
Install the correct reactor once at process start; changing reactors mid-process is unsupported.
Watch logs at startup: a NotConfigured handler is skipped silently and only fails later as unsupported scheme.

When it happens

Trigger: Setting DOWNLOAD_HANDLERS to a BaseStreamingDownloadHandler subclass while keeping the default (non-asyncio) Twisted reactor; running on Windows with the old default reactor; TWISTED_REACTOR explicitly set to a non-asyncio reactor such as 'twisted.internet.default:selectreactor'.

Common situations: Adopting the new streaming handlers without the required reactor migration; custom settings bundles (e.g. company baselines) that pin the default reactor; upgrading Scrapy where TWISTED_REACTOR defaults changed and overrides now conflict.

Related errors

AI-assisted analysis of scrapy/scrapy@06af687662 (2026-08-15). Data as JSON: /api/errors/451f802143a10836. Report an issue: GitHub.