{"record":{"id":"e7de78bda371f166","repo":"zylon-ai/private-gpt","slug":"web-fetch-markdown-cleaning-dependencies-are-not-i","errorCode":null,"errorMessage":"Web fetch markdown cleaning dependencies are not installed. Install with `uv sync --inexact --extra ingest-markup`.","messagePattern":"Web fetch markdown cleaning dependencies are not installed\\. Install with `uv sync --inexact --extra ingest-markup`\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/web/web_scraper_service.py","lineNumber":24,"sourceCode":"import time\nfrom typing import Any\n\nimport html2text  # ty:ignore[unresolved-import]\nfrom injector import inject, singleton\nfrom pydantic import BaseModel\n\nfrom private_gpt.components.web.scraper.registry import WebScraperProviderRegistry\nfrom private_gpt.settings.settings import Settings\nfrom private_gpt.utils.dependencies import format_missing_dependency_message\n\nlogger = logging.getLogger(__name__)\n\n\ndef _get_html_cleaner() -> Any:\n    try:\n        return importlib.import_module(\"lxml_html_clean\").Cleaner\n    except ImportError as e:\n        raise ImportError(\n            format_missing_dependency_message(\n                \"Web fetch markdown cleaning\",\n                extras=\"ingest-markup\",\n            )\n        ) from e\n\n\nclass WebScraperResult(BaseModel):\n    url: str | None = None\n    html_content: str | None = None\n    markdown_content: str | None = None\n    favicon_url: str | None = None\n\n\n@singleton\nclass WebScraperService:\n    @inject\n    def __init__(self, settings: Settings) -> None:","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/web/web_scraper_service.py#L6-L42","documentation":"ImportError raised lazily by _get_html_cleaner() when the lxml_html_clean package cannot be imported. private-gpt splits optional ingestion dependencies into extras, and HTML→markdown cleaning for web fetch lives in the ingest-markup extra; without it the cleaner dependency simply is not on sys.path.","triggerScenarios":"Installing with `uv sync` (exact, no extras) then calling WebScraperService.scrape() which converts HTML to markdown; a deployment image built without the ingest-markup extra; lxml 5+/html CLEANer split where the old lxml.html.clean module moved to the separate lxml_html_clean distribution.","commonSituations":"Minimal installs that only pull core deps; the upstream lxml change that extracted HTML cleaner into lxml-html-clean, which bites projects that previously relied on lxml alone; CI environments synced with a trimmed feature set.","solutions":["Install the extra exactly as the message says: `uv sync --inexact --extra ingest-markup`.","Or add `lxml-html-clean` explicitly to the deployment's requirements.","Rebuild container images after adding the extra so runtime environments match."],"exampleFix":"# before\nuv sync\n# raises ImportError: Web fetch markdown cleaning dependencies are not installed...\n\n# after\nuv sync --inexact --extra ingest-markup","handlingStrategy":"validation","validationCode":"def markup_deps_available() -> bool:\n    try:\n        import lxml_html_clean  # noqa: F401\n        return True\n    except ImportError:\n        return False\n\nif not markup_deps_available():\n    raise SystemExit('Install first: uv sync --inexact --extra ingest-markup')","typeGuard":"def is_missing_markup_extra(exc: BaseException) -> bool:\n    return isinstance(exc, ImportError) and 'ingest-markup' in str(exc)","tryCatchPattern":"try:\n    result = await scraper.scrape(url)\nexcept ImportError as e:\n    if 'ingest-markup' in str(e):\n        raise SystemExit('Run: uv sync --inexact --extra ingest-markup') from e\n    raise","preventionTips":["Document which extras each feature needs in deployment READMEs.","Add a startup dependency probe that fails fast listing missing extras.","Use the same uv sync command (with extras) in CI, Docker, and local dev."],"tags":["dependencies","import","install","extras"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}