assafelovic/gpt-researcher · error · ImportError

Unable to install {pkg_inst_name}. Please install manually w

Error message

Unable to install {pkg_inst_name}. Please install manually with `pip install -U {pkg_inst_name}`

What it means

Raised by gpt_researcher's Scraper __init__ when it attempts to auto-install an optional scraper dependency via `pip install` and the subprocess fails (CalledProcessError). The library tries to bootstrap missing packages on demand, and only raises this ImportError when that automatic install fails. The message names the exact package so the user can install it manually.

Source

Thrown at gpt_researcher/scraper/scraper.py:188

            },
            "firecrawl": {
                "package_installation_name": "firecrawl-py",
                "import_name": "firecrawl",
            },
        }
        pkg = pkg_map[scrapper_name]
        if not importlib.util.find_spec(pkg["import_name"]):
            pkg_inst_name = pkg["package_installation_name"]
            init(autoreset=True)
            print(Fore.YELLOW + f"{pkg_inst_name} not found. Attempting to install...")
            try:
                subprocess.check_call(
                    [sys.executable, "-m", "pip", "install", pkg_inst_name]
                )
                importlib.invalidate_caches()
                print(Fore.GREEN + f"{pkg_inst_name} installed successfully.")
            except subprocess.CalledProcessError:
                raise ImportError(
                    Fore.RED
                    + f"Unable to install {pkg_inst_name}. Please install manually with "
                    f"`pip install -U {pkg_inst_name}`"
                )

    async def extract_data_from_url(self, link, session):
        """
        Extracts the data from the link with logging
        """
        async with self.worker_pool.throttle():
            try:
                # Reject SSRF / local-file targets (internal hosts, cloud metadata
                # endpoints, file:// paths, etc.) before any request is made.
                try:
                    validate_url(link)
                except UnsafeURLError as e:
                    self.logger.warning(f"Skipping unsafe URL {link}: {e}")
                    return {

View on GitHub (pinned to 6f998577d5)

Solutions

Install the named package manually: pip install -U <pkg_inst_name>
If in a container/CI, add the package to the image's requirements and rebuild
For externally-managed environments (Debian/Ubuntu system Python), use a virtualenv or pass --break-system-packages
Verify pip works for the same interpreter: python -m pip --version

Example fix

# before
from gpt_researcher.scraper.scraper import Scraper
s = Scraper('beautifulsoup')  # ImportError: Unable to install bs4...

# after
pip install -U beautifulsoup4 lxml
# then
from gpt_researcher.scraper.scraper import Scraper
s = Scraper('beautifulsoup')

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util, sys

def scraper_pkg_available(mod_name: str) -> bool:
    return importlib.util.find_spec(mod_name) is not None

if not scraper_pkg_available('fitz'):  # pymupdf
    raise SystemExit('Install pymupdf first: pip install -U pymupdf')

Try / catch

try:
    scraper = Scraper('pymupdf')
except ImportError as e:
    # fall back to a dependency-free scraper
    scraper = Scraper('beautifulsoup')

Prevention

Pin all optional scraper deps in requirements.txt so pip never needs to self-install at runtime
Run scraper containers with network access during build, not at runtime
Pre-install known scraper packages in Dockerfiles: pip install beautifulsoup4 pymupdf lxml

When it happens

Trigger: Instantiating a scraper class (e.g. BeautifulSoupScraper, PyMuPDFScraper) whose backing package (beautifulsoup4, pymupdf, etc.) is missing from the environment AND pip install of that package fails — e.g. no network, read-only site-packages, pip not on PATH for sys.executable, or an incompatible Python version for the wheel.

Common situations: Running in a locked-down Docker/CI container without network egress, using a system Python where pip install is blocked (PEP 668 externally-managed environment), offline air-gapped installs, or Python versions with no prebuilt wheels for the scraper package.

Related errors

Unable to import {pkg_kebab}. Please install with `pip insta

AI-assisted analysis of assafelovic/gpt-researcher@6f998577d5 (2026-08-28). Data as JSON: /api/errors/6a2d7719ca183a25. Report an issue: GitHub.