zylon-ai/private-gpt · error · ImportError

Web fetch markdown cleaning dependencies are not installed.

Error message

Web fetch markdown cleaning dependencies are not installed. Install with `uv sync --inexact --extra ingest-markup`.

What it means

ImportError raised lazily by _get_html_cleaner() when the lxml_html_clean package cannot be imported. private-gpt splits optional ingestion dependencies into extras, and HTML→markdown cleaning for web fetch lives in the ingest-markup extra; without it the cleaner dependency simply is not on sys.path.

Source

Thrown at private_gpt/components/web/web_scraper_service.py:24

import time
from typing import Any

import html2text  # ty:ignore[unresolved-import]
from injector import inject, singleton
from pydantic import BaseModel

from private_gpt.components.web.scraper.registry import WebScraperProviderRegistry
from private_gpt.settings.settings import Settings
from private_gpt.utils.dependencies import format_missing_dependency_message

logger = logging.getLogger(__name__)


def _get_html_cleaner() -> Any:
    try:
        return importlib.import_module("lxml_html_clean").Cleaner
    except ImportError as e:
        raise ImportError(
            format_missing_dependency_message(
                "Web fetch markdown cleaning",
                extras="ingest-markup",
            )
        ) from e


class WebScraperResult(BaseModel):
    url: str | None = None
    html_content: str | None = None
    markdown_content: str | None = None
    favicon_url: str | None = None


@singleton
class WebScraperService:
    @inject
    def __init__(self, settings: Settings) -> None:

View on GitHub (pinned to 4a030776a3)

Solutions

Install the extra exactly as the message says: `uv sync --inexact --extra ingest-markup`.
Or add `lxml-html-clean` explicitly to the deployment's requirements.
Rebuild container images after adding the extra so runtime environments match.

Example fix

# before
uv sync
# raises ImportError: Web fetch markdown cleaning dependencies are not installed...

# after
uv sync --inexact --extra ingest-markup

Defensive patterns

Strategy: validation

Validate before calling

def markup_deps_available() -> bool:
    try:
        import lxml_html_clean  # noqa: F401
        return True
    except ImportError:
        return False

if not markup_deps_available():
    raise SystemExit('Install first: uv sync --inexact --extra ingest-markup')

Type guard

def is_missing_markup_extra(exc: BaseException) -> bool:
    return isinstance(exc, ImportError) and 'ingest-markup' in str(exc)

Try / catch

try:
    result = await scraper.scrape(url)
except ImportError as e:
    if 'ingest-markup' in str(e):
        raise SystemExit('Run: uv sync --inexact --extra ingest-markup') from e
    raise

Prevention

Document which extras each feature needs in deployment READMEs.
Add a startup dependency probe that fails fast listing missing extras.
Use the same uv sync command (with extras) in CI, Docker, and local dev.

When it happens

Trigger: Installing with `uv sync` (exact, no extras) then calling WebScraperService.scrape() which converts HTML to markdown; a deployment image built without the ingest-markup extra; lxml 5+/html CLEANer split where the old lxml.html.clean module moved to the separate lxml_html_clean distribution.

Common situations: Minimal installs that only pull core deps; the upstream lxml change that extracted HTML cleaner into lxml-html-clean, which bites projects that previously relied on lxml alone; CI environments synced with a trimmed feature set.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/e7de78bda371f166. Report an issue: GitHub.