scrapy/scrapy · critical · KeyError

Spider not found: {spider_name}

Error message

Spider not found: {spider_name}

What it means

Raised by SpiderLoader.load(spider_name) when the name is not in the loaded spider registry. The loader builds {name: SpiderClass} from the SPIDER_MODULES setting at startup; an unknown name re-raises as KeyError('Spider not found: ...') with the original KeyError suppressed. This is the error behind 'scrapy crawl unknown-name' style failures.

Source

Thrown at scrapy/spiderloader.py:125

    @classmethod
    def from_settings(cls, settings: BaseSettings) -> Self:
        """Create an instance of the class.

        It's called with the current project settings, and it loads the spiders
        found recursively in the modules of the :setting:`SPIDER_MODULES`
        setting.
        """
        return cls(settings)

    def load(self, spider_name: str) -> type[Spider]:
        """Return the spider class for the given spider name.

        If the spider name is not found, raise a :exc:`KeyError`.
        """
        try:
            return self._spiders[spider_name]
        except KeyError:
            raise KeyError(f"Spider not found: {spider_name}") from None

    def find_by_request(self, request: Request) -> list[str]:
        """
        Return the list of spider names that can handle the given request.

        It will try to match the request's url against the domains of
        the spiders.
        """
        return [
            name for name, cls in self._spiders.items() if cls.handles_request(request)
        ]

    def list(self) -> list[str]:
        """Return a list with the names of all spiders available in the project."""
        return list(self._spiders.keys())


class DummySpiderLoader:

View on GitHub (pinned to 06af687662)

Solutions

1. List what is actually available: print(spider_loader.list()) and use one of those names
2. Ensure SPIDER_MODULES in settings.py contains the package holding your spider class
3. Fix import/syntax errors in the spider module and confirm the class defines name = 'my_spider'
4. Select by class instead: CrawlerProcess.crawl(MySpider) avoids name lookup

Example fix

# before
process.crawl('my_spider')  # KeyError: Spider not found: my_spider

# after
from myproject.spiders.my import MySpider
process.crawl(MySpider)

names = spider_loader.list()
if spider_name not in names:
    raise SystemExit(f'unknown spider {spider_name!r}; available: {sorted(names)}')

try:
    spider_cls = spider_loader.load(name)
except KeyError as e:
    print(f'available spiders: {spider_loader.list()}')
    raise

When it happens

Trigger: spider_loader.load('my_spider') where 'my_spider' is not defined: spider module not listed in SPIDER_MODULES, class has no name attribute or a different name, spider file has syntax/import errors (so it never registered), or a typo in the name.

Common situations: Running crawls from scripts with CrawlerProcess and a mistyped name; moving spiders without updating SPIDER_MODULES; a broken import inside a spider module silently dropping it from the registry; name collisions causing unexpected names.

Related errors

• DummySpiderLoader doesn't load any spiders
• {type(self).__name__} must have a name
• Unsupported node iterator
• Invalid entry point {entry_point.name}
• crawler already set

AI-assisted analysis of scrapy/scrapy@06af687662 (2026-08-15). Data as JSON: /api/errors/6c6c02bce0614844. Report an issue: GitHub.