{"record":{"id":"6a2d7719ca183a25","repo":"assafelovic/gpt-researcher","slug":"unable-to-install-pkg-inst-name-please-install","errorCode":null,"errorMessage":"Unable to install {pkg_inst_name}. Please install manually with `pip install -U {pkg_inst_name}`","messagePattern":"Unable to install (.+?)\\. Please install manually with `pip install -U (.+?)`","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"gpt_researcher/scraper/scraper.py","lineNumber":188,"sourceCode":"            },\n            \"firecrawl\": {\n                \"package_installation_name\": \"firecrawl-py\",\n                \"import_name\": \"firecrawl\",\n            },\n        }\n        pkg = pkg_map[scrapper_name]\n        if not importlib.util.find_spec(pkg[\"import_name\"]):\n            pkg_inst_name = pkg[\"package_installation_name\"]\n            init(autoreset=True)\n            print(Fore.YELLOW + f\"{pkg_inst_name} not found. Attempting to install...\")\n            try:\n                subprocess.check_call(\n                    [sys.executable, \"-m\", \"pip\", \"install\", pkg_inst_name]\n                )\n                importlib.invalidate_caches()\n                print(Fore.GREEN + f\"{pkg_inst_name} installed successfully.\")\n            except subprocess.CalledProcessError:\n                raise ImportError(\n                    Fore.RED\n                    + f\"Unable to install {pkg_inst_name}. Please install manually with \"\n                    f\"`pip install -U {pkg_inst_name}`\"\n                )\n\n    async def extract_data_from_url(self, link, session):\n        \"\"\"\n        Extracts the data from the link with logging\n        \"\"\"\n        async with self.worker_pool.throttle():\n            try:\n                # Reject SSRF / local-file targets (internal hosts, cloud metadata\n                # endpoints, file:// paths, etc.) before any request is made.\n                try:\n                    validate_url(link)\n                except UnsafeURLError as e:\n                    self.logger.warning(f\"Skipping unsafe URL {link}: {e}\")\n                    return {","sourceCodeStart":170,"sourceCodeEnd":206,"githubUrl":"https://github.com/assafelovic/gpt-researcher/blob/6f998577d547b1e54ec662dac63583aa11e3b84b/gpt_researcher/scraper/scraper.py#L170-L206","documentation":"Raised by gpt_researcher's Scraper __init__ when it attempts to auto-install an optional scraper dependency via `pip install` and the subprocess fails (CalledProcessError). The library tries to bootstrap missing packages on demand, and only raises this ImportError when that automatic install fails. The message names the exact package so the user can install it manually.","triggerScenarios":"Instantiating a scraper class (e.g. BeautifulSoupScraper, PyMuPDFScraper) whose backing package (beautifulsoup4, pymupdf, etc.) is missing from the environment AND pip install of that package fails — e.g. no network, read-only site-packages, pip not on PATH for sys.executable, or an incompatible Python version for the wheel.","commonSituations":"Running in a locked-down Docker/CI container without network egress, using a system Python where pip install is blocked (PEP 668 externally-managed environment), offline air-gapped installs, or Python versions with no prebuilt wheels for the scraper package.","solutions":["Install the named package manually: pip install -U <pkg_inst_name>","If in a container/CI, add the package to the image's requirements and rebuild","For externally-managed environments (Debian/Ubuntu system Python), use a virtualenv or pass --break-system-packages","Verify pip works for the same interpreter: python -m pip --version"],"exampleFix":"# before\nfrom gpt_researcher.scraper.scraper import Scraper\ns = Scraper('beautifulsoup')  # ImportError: Unable to install bs4...\n\n# after\npip install -U beautifulsoup4 lxml\n# then\nfrom gpt_researcher.scraper.scraper import Scraper\ns = Scraper('beautifulsoup')","handlingStrategy":"validation","validationCode":"import importlib.util, sys\n\ndef scraper_pkg_available(mod_name: str) -> bool:\n    return importlib.util.find_spec(mod_name) is not None\n\nif not scraper_pkg_available('fitz'):  # pymupdf\n    raise SystemExit('Install pymupdf first: pip install -U pymupdf')","typeGuard":null,"tryCatchPattern":"try:\n    scraper = Scraper('pymupdf')\nexcept ImportError as e:\n    # fall back to a dependency-free scraper\n    scraper = Scraper('beautifulsoup')","preventionTips":["Pin all optional scraper deps in requirements.txt so pip never needs to self-install at runtime","Run scraper containers with network access during build, not at runtime","Pre-install known scraper packages in Dockerfiles: pip install beautifulsoup4 pymupdf lxml"],"tags":["pip","dependency-install","packaging","subprocess"],"backgroundTag":"pip-install-failed","analyzedSha":"6f998577d547b1e54ec662dac63583aa11e3b84b","analyzedAt":"2026-08-28T17:50:07.383Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}