D4Vinci/Scrapling · error · ModuleNotFoundError

You need to install scrapling with any of the extras to enab

Error message

You need to install scrapling with any of the extras to enable Shell commands. See: https://scrapling.readthedocs.io/en/latest/#installation

What it means

scrapling.cli imports click at module load; click is only pulled in by the extras (shell/CLI extras), not by a minimal `pip install scrapling`. If click is missing, the import failure is re-raised as a ModuleNotFoundError pointing at the install docs so the user knows the CLI is opt-in. This fires the moment anything imports scrapling.cli (typically running the `scrapling` console script).

Source

Thrown at scrapling/cli.py:17

from os import environ
from pathlib import Path
from subprocess import check_output
from sys import executable as python_executable

from scrapling import __version__
from scrapling.core.utils import log
from scrapling.engines.toolbelt.custom import Response
from scrapling.core.utils._shell import _CookieParser, _ParseHeaders
from scrapling.core._types import List, Optional, Dict, Tuple, Any, Callable

from orjson import loads as json_loads, JSONDecodeError

try:
    from click import command, option, Choice, group, argument, version_option
except (ImportError, ModuleNotFoundError) as e:
    raise ModuleNotFoundError(
        "You need to install scrapling with any of the extras to enable Shell commands. See: https://scrapling.readthedocs.io/en/latest/#installation"
    ) from e

__OUTPUT_FILE_HELP__ = "The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively."
__PACKAGE_DIR__ = Path(__file__).parent


def __Execute(cmd: List[str], help_line: str) -> None:  # pragma: no cover
    print(f"Installing {help_line}...")
    _ = check_output(cmd, shell=False)  # nosec B603
    # I meant to not use try except here


def __ParseJSONData(json_string: Optional[str] = None) -> Optional[Dict[str, Any]]:
    """Parse JSON string into a Python object"""
    if not json_string:
        return None

View on GitHub (pinned to 5d213a2d47)

Solutions

1. Install with the extras that include the shell/CLI tooling, e.g. `pip install "scrapling[all]"` (or the shell-specific extra shown in the linked docs)
2. Or install click directly: `pip install click`
3. Verify you are running the CLI with the same interpreter/venv where scrapling+extras are installed (`which scrapling`, `pip show click`)
4. If you never need the CLI, avoid importing scrapling.cli (programmatic fetchers do not require click)

Example fix

# before
pip install scrapling
scrapling fetch https://example.com
# ModuleNotFoundError: You need to install scrapling with any of the extras...

# after
pip install "scrapling[all]"
scrapling fetch https://example.com

import importlib.util, sys

if importlib.util.find_spec('click') is None:
    sys.exit('scrapling CLI needs extras: pip install "scrapling[all]"')

When it happens

Trigger: Running `scrapling --version` or `scrapling fetch ...` after installing scrapling without extras (no click on the environment). Also importing scrapling.core.utils._shell helpers transitively in an environment where click was pruned or a different venv/interpreter is active.

Common situations: Minimal installs in slim Docker images, CI caches that dropped extras, upgrading scrapling in a venv created with `--no-deps`, or using a system Python where the package script points to an interpreter without click.

Related errors

• Invalid JSON data '{json_string}': {err}
• Could not parse cookies '{cookies}': {err}
• Could not parse header without colon: '{header_line}'.
• Could not parse cookie string from header '{header_value}':

AI-assisted analysis of D4Vinci/Scrapling@5d213a2d47 (2026-08-14). Data as JSON: /api/errors/b26abf7657243120. Report an issue: GitHub.