zylon-ai/private-gpt · error · RuntimeError

Scraper service not initialized

Error message

Scraper service not initialized

What it means

RuntimeError from ScrapedContentProcessor.process_results when _scraper_service is still None. The processor resolves WebScraperService lazily via get_global_injector().get(...) inside _initialize(); the guard fires only when that DI lookup failed or the processor was constructed outside the injector's lifecycle.

Source

Thrown at private_gpt/components/web/web_search/processors/scraped_content_processor.py:38

        settings: Settings,
    ):
        super().__init__()
        self._settings = settings
        self._scraper_service = None
        self._initialize()

    def _initialize(self) -> None:
        if self._scraper_service is None:
            self._scraper_service = get_global_injector().get(WebScraperService)

    async def process_results(
        self,
        query: str,
        results: list[WebSearchResult],
        model_id: str | None = None,
    ) -> list[WebSearchResult]:
        if self._scraper_service is None:
            raise RuntimeError("Scraper service not initialized")

        limited_results = results[0 : self._settings.web_search.num_links]

        tasks = [self._scraper_service.scrape(result.url) for result in limited_results]
        scraped_contents = await asyncio.gather(*tasks, return_exceptions=True)

        for idx, (result, scraped_content) in enumerate(
            zip(limited_results, scraped_contents, strict=False), 1
        ):
            result.idx = idx

            if isinstance(scraped_content, Exception):
                logger.warning(
                    f"ScrapedContentProcessor: Failed to scrape {result.url}: {scraped_content}"
                )
                result.content = f"Failed to scrape content: {scraped_content!s}"
                result.is_in_error = True
            else:

View on GitHub (pinned to 4a030776a3)

Solutions

1. Obtain the processor through the DI container (get_global_injector().get) or via WebSearchService so dependencies are injected.
2. In tests, configure the injector / inject a fake WebScraperService instead of constructing the processor bare.
3. Check earlier logs for an exception during get_global_injector().get(WebScraperService) — the None here is usually downstream of that failure.

Example fix

# before (test)
proc = ScrapedContentProcessor(settings)
# _initialize failed silently, later RuntimeError

# after (test)
injector = Injector([SettingsModule(settings)])
service = injector.get(WebScraperService)
proc = ScrapedContentProcessor(settings)
proc._scraper_service = service

# ensure DI is wired before using the processor
from private_gpt.di import get_global_injector
from private_gpt.components.web.web_scraper_service import WebScraperService

svc = get_global_injector().get(WebScraperService)  # raises loudly if unbindable
processor = get_global_injector().get(WebSearchService)._processor

def processor_ready(proc: ScrapedContentProcessor) -> bool:
    return proc._scraper_service is not None

try:
    results = await processor.process_results(query, results)
except RuntimeError as e:
    if 'Scraper service not initialized' in str(e):
        raise RuntimeError('DI misconfiguration: resolve via injector') from e
    raise

When it happens

Trigger: Instantiating ScrapedContentProcessor manually in tests without configuring the global injector; the injector raising during .get(WebScraperService) being swallowed upstream; importing and using the processor before the private-gpt DI container is built.

Common situations: Unit tests that new-up the processor with only settings; scripts that bypass private_gpt.di; circular-import or initialization-order issues where the processor is created during injector bootstrap before WebScraperService is bound.

Related errors

• Web Search is not properly initialized or is disabled in set
• No default model configured to set LLM
• Names already registered: {existing_names}
• API base URL and poll interval must be provided in async mod
• Sandbox client not configured

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/dfa7be69b34d4b11. Report an issue: GitHub.