scrapy/scrapy · error · NotImplementedError

MiddlewareManager.open_spider() is no longer implemented and

Error message

MiddlewareManager.open_spider() is no longer implemented and will be removed in a future Scrapy version.

What it means

MiddlewareManager.open_spider() now unconditionally raises NotImplementedError. Spider open/close hooks were removed from the middleware manager API in recent Scrapy versions; middleware from_opening/start/finish handling moved elsewhere (e.g., from_crawler and engine-managed signals). The method stub exists only to break loudly for callers still using the old API.

Source

Thrown at scrapy/middleware.py:156

        warn_deferred: bool = False,
    ) -> _T:
        methods = cast(
            "Iterable[Callable[Concatenate[_T, _P], _T]]", self.methods[methodname]
        )
        for method in methods:
            warn = global_object_name(method) if warn_deferred else None
            if always_add_spider or (
                add_spider and method in self._mw_methods_requiring_spider
            ):
                obj = await ensure_awaitable(
                    method(obj, *(*args, self._spider)), _warn=warn
                )
            else:
                obj = await ensure_awaitable(method(obj, *args), _warn=warn)
        return obj

    def open_spider(self, spider: Spider) -> Deferred[list[None]]:  # pragma: no cover
        raise NotImplementedError(
            "MiddlewareManager.open_spider() is no longer implemented"
            " and will be removed in a future Scrapy version."
        )

    def close_spider(self, spider: Spider) -> Deferred[list[None]]:  # pragma: no cover
        raise NotImplementedError(
            "MiddlewareManager.close_spider() is no longer implemented"
            " and will be removed in a future Scrapy version."
        )

View on GitHub (pinned to 06af687662)

Solutions

1. Remove calls to MiddlewareManager.open_spider(); rely on from_crawler and the engine's spider lifecycle.
2. Use the spider_opened signal for setup that used to happen in open_spider.
3. If subclassing, delete overridden open_spider implementations rather than calling super().

Example fix

# before
manager.open_spider(spider)  # NotImplementedError

# after
from scrapy import signals
crawler.signals.connect(self._on_open, signal=signals.spider_opened)

import inspect
from scrapy.middleware import MiddlewareManager

assert not type(manager).open_spider.__qualname__.startswith('MiddlewareManager'), \
    'open_spider removed; use signals'

When it happens

Trigger: Calling manager.open_spider(spider) on any MiddlewareManager (SpiderMiddlewareManager or DownloaderMiddlewareManager); third-party code or tests that orchestrated crawls by manually calling open_spider.

Common situations: Upgrading Scrapy major versions where open_spider used to return a Deferred firing when middlewares opened; custom crawl runners built around the old manager lifecycle; stale tutorials.

Related errors

• MiddlewareManager.close_spider() is no longer implemented an
• {global_object_name(type(self._spider))}.start() must be an
• Middleware {global_object_name(mw.__class__)} doesn't suppor
• Crawler.get_downloader_middleware() can only be called after
• Crawler.get_spider_middleware() can only be called after the

AI-assisted analysis of scrapy/scrapy@06af687662 (2026-08-15). Data as JSON: /api/errors/89545497f0d1726f. Report an issue: GitHub.