zylon-ai/private-gpt · error · ValueError

Web fetching is not properly initialized or it is disabled.

Error message

Web fetching is not properly initialized or it is disabled. Since Web Search depends on web fetching to retrieve content, the Web Search functionality cannot operate correctly. Consider enabling web fetching in settings.

What it means

ValueError from CleanContentProcessor.validate(): the web search pipeline was asked to fetch and clean result pages, but the shared WebScraperService reports is_initialized == False, meaning settings.web_fetch.enabled was false at startup. Web search's clean_content processor depends on web fetching to retrieve page bodies, so validation aborts before any query runs.

Source

Thrown at private_gpt/components/web/web_search/processors/clean_content.py:34

    )


logger = logging.getLogger(__name__)


class CleanContentProcessor(BaseWebSearchResultProcessor):
    def __init__(
        self,
        settings: Settings,
        scraper_service: WebScraperService,
    ) -> None:
        super().__init__()
        self._settings = settings
        self._scraper_service = scraper_service

    async def validate(self) -> None:
        if not self._scraper_service.is_initialized:
            raise ValueError(
                "Web fetching is not properly initialized or it is disabled. "
                "Since Web Search depends on web fetching to retrieve content, "
                "the Web Search functionality cannot operate correctly. "
                "Consider enabling web fetching in settings."
            )

    async def process_results(
        self,
        query: str,
        results: list[WebSearchResult],
        model_id: str | None = None,
    ) -> list[WebSearchResult]:
        processed_results: list[WebSearchResult] = []
        for result in results[0 : self._settings.web_search.num_links]:
            try:
                # Get HTML for the result - lazy load scraper service
                response: WebScraperResult = (
                    await self._scraper_service.scrape_max_compress(result.url)

View on GitHub (pinned to 4a030776a3)

Solutions

Enable web fetching: web_fetch.enabled: true alongside web_search.enabled: true.
If you do not want page fetching, switch web_search.processor to simple_text which has no scraper dependency.
Restart the service after settings changes — initialization flags are set once at startup.

Example fix

# settings.yaml
web_search:
  enabled: true
  processor: clean_content
web_fetch:
  enabled: true  # required by clean_content

Defensive patterns

Strategy: validation

Validate before calling

async def web_search_ready(search_svc: WebSearchService,
                              scraper_svc: WebScraperService) -> bool:
    return (
        search_svc._initialized
        and scraper_svc.is_initialized  # processor's requirement
    )

Try / catch

try:
    results = await search_svc.search(query)
except ValueError as e:
    if 'Web Search depends on web fetching' in str(e):
        raise RuntimeError('Config error: enable web_fetch for clean_content processor') from e
    raise

Prevention

Choose simple_text processor when web fetching is intentionally disabled.
Run validate() in a health endpoint so misconfig is visible before user traffic.
Encapsulate 'web search requires web fetch' in deployment docs and config examples.

When it happens

Trigger: web_search.enabled: true with processor: clean_content but web_fetch.enabled left false (its default); calling WebSearchService.search() which first awaits validate() → processor.validate().

Common situations: Enabling web search without realizing its content processors require the separate web-fetch flag; config split across profiles where the fetch toggle lives in a profile that is not active.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/278aaa901984ff80. Report an issue: GitHub.