{"record":{"id":"ff1e3b1cef8ea246","repo":"assafelovic/gpt-researcher","slug":"scraper-not-found","errorCode":null,"errorMessage":"Scraper not found.","messagePattern":"Scraper not found\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"gpt_researcher/scraper/scraper.py","lineNumber":350,"sourceCode":"            \"firecrawl\": FireCrawl,\n        }\n\n        scraper_key = None\n\n        # Inspect only the path component so query strings / fragments don't\n        # hide the extension (e.g. signed CDN/S3 links like \"…/doc.pdf?sig=…\").\n        # Match case-insensitively because \".PDF\" is a perfectly valid suffix.\n        path = urlparse(link).path\n        if path.lower().endswith(\".pdf\"):\n            scraper_key = \"pdf\"\n        elif \"arxiv.org\" in link:\n            scraper_key = \"arxiv\"\n        else:\n            scraper_key = self.scraper\n\n        scraper_class = SCRAPER_CLASSES.get(scraper_key)\n        if scraper_class is None:\n            raise Exception(\"Scraper not found.\")\n\n        return scraper_class\n","sourceCodeStart":332,"sourceCodeEnd":353,"githubUrl":"https://github.com/assafelovic/gpt-researcher/blob/6f998577d547b1e54ec662dac63583aa11e3b84b/gpt_researcher/scraper/scraper.py#L332-L353","documentation":"Raised by Scraper.get_scraper when the requested scraper key has no mapping in SCRAPER_CLASSES. The class only supports a fixed registry of scrapers (beautifulsoup, arxiv, pymupdf/pdf, tavily_extract, etc.), and any other string — or a typo — falls through to this generic Exception.","triggerScenarios":"Calling extract_data_from_url (or get_scraper directly) with a scraper name not in SCRAPER_CLASSES, e.g. Scraper('bs4'), Scraper('BeautifulSoup'), Scraper('playwright'), or when URL-sniffing selects a key that isn't registered.","commonSituations":"Typos or wrong casing in the doc_source/scraper config value; expecting a scraper (e.g. playwright or selenium) that the installed gpt-researcher version doesn't register; using a custom scraper name without registering it.","solutions":["Check SCRAPER_CLASSES keys in gpt_researcher/scraper/scraper.py and use an exact key like 'beautifulsoup'","Fix the config value (doc_source / scraper setting) that feeds Scraper(...)","Upgrade gpt-researcher — newer versions register more scrapers","For custom scrapers, register your class in SCRAPER_CLASSES via subclassing/monkeypatching"],"exampleFix":"# before\nscraper = Scraper('bs4')  # Exception: Scraper not found.\n\n# after\nscraper = Scraper('beautifulsoup')","handlingStrategy":"validation","validationCode":"from gpt_researcher.scraper.scraper import SCRAPER_CLASSES\n\nname = 'bs4'\nif name not in SCRAPER_CLASSES:\n    raise ValueError(f'Unknown scraper {name!r}; valid: {sorted(SCRAPER_CLASSES)}')\nscraper = Scraper(name)","typeGuard":null,"tryCatchPattern":"try:\n    scraper_class = Scraper('x').get_scraper()\nexcept Exception as e:\n    if 'Scraper not found' in str(e):\n        scraper_class = Scraper('beautifulsoup').get_scraper()\n    else:\n        raise","preventionTips":["Read valid keys from SCRAPER_CLASSES instead of hardcoding names","Validate user-supplied doc_source values against the registry before constructing Scraper","Log the registry keys on startup so misconfiguration is obvious"],"tags":["scraper","registry","invalid-argument","lookup"],"backgroundTag":"unsupported-scraper-backend","analyzedSha":"6f998577d547b1e54ec662dac63583aa11e3b84b","analyzedAt":"2026-08-28T17:50:07.383Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}