assafelovic/gpt-researcher · error · Exception

Scraper not found.

Error message

Scraper not found.

What it means

Raised by Scraper.get_scraper when the requested scraper key has no mapping in SCRAPER_CLASSES. The class only supports a fixed registry of scrapers (beautifulsoup, arxiv, pymupdf/pdf, tavily_extract, etc.), and any other string — or a typo — falls through to this generic Exception.

Source

Thrown at gpt_researcher/scraper/scraper.py:350

            "firecrawl": FireCrawl,
        }

        scraper_key = None

        # Inspect only the path component so query strings / fragments don't
        # hide the extension (e.g. signed CDN/S3 links like "…/doc.pdf?sig=…").
        # Match case-insensitively because ".PDF" is a perfectly valid suffix.
        path = urlparse(link).path
        if path.lower().endswith(".pdf"):
            scraper_key = "pdf"
        elif "arxiv.org" in link:
            scraper_key = "arxiv"
        else:
            scraper_key = self.scraper

        scraper_class = SCRAPER_CLASSES.get(scraper_key)
        if scraper_class is None:
            raise Exception("Scraper not found.")

        return scraper_class

View on GitHub (pinned to 6f998577d5)

Solutions

1. Check SCRAPER_CLASSES keys in gpt_researcher/scraper/scraper.py and use an exact key like 'beautifulsoup'
2. Fix the config value (doc_source / scraper setting) that feeds Scraper(...)
3. Upgrade gpt-researcher — newer versions register more scrapers
4. For custom scrapers, register your class in SCRAPER_CLASSES via subclassing/monkeypatching

Example fix

# before
scraper = Scraper('bs4')  # Exception: Scraper not found.

# after
scraper = Scraper('beautifulsoup')

from gpt_researcher.scraper.scraper import SCRAPER_CLASSES

name = 'bs4'
if name not in SCRAPER_CLASSES:
    raise ValueError(f'Unknown scraper {name!r}; valid: {sorted(SCRAPER_CLASSES)}')
scraper = Scraper(name)

try:
    scraper_class = Scraper('x').get_scraper()
except Exception as e:
    if 'Scraper not found' in str(e):
        scraper_class = Scraper('beautifulsoup').get_scraper()
    else:
        raise

When it happens

Trigger: Calling extract_data_from_url (or get_scraper directly) with a scraper name not in SCRAPER_CLASSES, e.g. Scraper('bs4'), Scraper('BeautifulSoup'), Scraper('playwright'), or when URL-sniffing selects a key that isn't registered.

Common situations: Typos or wrong casing in the doc_source/scraper config value; expecting a scraper (e.g. playwright or selenium) that the installed gpt-researcher version doesn't register; using a custom scraper name without registering it.

Related errors

• FireCrawl API key not found. Please set the FIRECRAWL_API_KE
• Model cannot be None
• URL must be a non-empty string.

AI-assisted analysis of assafelovic/gpt-researcher@6f998577d5 (2026-08-28). Data as JSON: /api/errors/ff1e3b1cef8ea246. Report an issue: GitHub.