{"record":{"id":"dfa7be69b34d4b11","repo":"zylon-ai/private-gpt","slug":"scraper-service-not-initialized","errorCode":null,"errorMessage":"Scraper service not initialized","messagePattern":"Scraper service not initialized","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/web/web_search/processors/scraped_content_processor.py","lineNumber":38,"sourceCode":"        settings: Settings,\n    ):\n        super().__init__()\n        self._settings = settings\n        self._scraper_service = None\n        self._initialize()\n\n    def _initialize(self) -> None:\n        if self._scraper_service is None:\n            self._scraper_service = get_global_injector().get(WebScraperService)\n\n    async def process_results(\n        self,\n        query: str,\n        results: list[WebSearchResult],\n        model_id: str | None = None,\n    ) -> list[WebSearchResult]:\n        if self._scraper_service is None:\n            raise RuntimeError(\"Scraper service not initialized\")\n\n        limited_results = results[0 : self._settings.web_search.num_links]\n\n        tasks = [self._scraper_service.scrape(result.url) for result in limited_results]\n        scraped_contents = await asyncio.gather(*tasks, return_exceptions=True)\n\n        for idx, (result, scraped_content) in enumerate(\n            zip(limited_results, scraped_contents, strict=False), 1\n        ):\n            result.idx = idx\n\n            if isinstance(scraped_content, Exception):\n                logger.warning(\n                    f\"ScrapedContentProcessor: Failed to scrape {result.url}: {scraped_content}\"\n                )\n                result.content = f\"Failed to scrape content: {scraped_content!s}\"\n                result.is_in_error = True\n            else:","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/web/web_search/processors/scraped_content_processor.py#L20-L56","documentation":"RuntimeError from ScrapedContentProcessor.process_results when _scraper_service is still None. The processor resolves WebScraperService lazily via get_global_injector().get(...) inside _initialize(); the guard fires only when that DI lookup failed or the processor was constructed outside the injector's lifecycle.","triggerScenarios":"Instantiating ScrapedContentProcessor manually in tests without configuring the global injector; the injector raising during .get(WebScraperService) being swallowed upstream; importing and using the processor before the private-gpt DI container is built.","commonSituations":"Unit tests that new-up the processor with only settings; scripts that bypass private_gpt.di; circular-import or initialization-order issues where the processor is created during injector bootstrap before WebScraperService is bound.","solutions":["Obtain the processor through the DI container (get_global_injector().get) or via WebSearchService so dependencies are injected.","In tests, configure the injector / inject a fake WebScraperService instead of constructing the processor bare.","Check earlier logs for an exception during get_global_injector().get(WebScraperService) — the None here is usually downstream of that failure."],"exampleFix":"# before (test)\nproc = ScrapedContentProcessor(settings)\n# _initialize failed silently, later RuntimeError\n\n# after (test)\ninjector = Injector([SettingsModule(settings)])\nservice = injector.get(WebScraperService)\nproc = ScrapedContentProcessor(settings)\nproc._scraper_service = service","handlingStrategy":"validation","validationCode":"# ensure DI is wired before using the processor\nfrom private_gpt.di import get_global_injector\nfrom private_gpt.components.web.web_scraper_service import WebScraperService\n\nsvc = get_global_injector().get(WebScraperService)  # raises loudly if unbindable\nprocessor = get_global_injector().get(WebSearchService)._processor","typeGuard":"def processor_ready(proc: ScrapedContentProcessor) -> bool:\n    return proc._scraper_service is not None","tryCatchPattern":"try:\n    results = await processor.process_results(query, results)\nexcept RuntimeError as e:\n    if 'Scraper service not initialized' in str(e):\n        raise RuntimeError('DI misconfiguration: resolve via injector') from e\n    raise","preventionTips":["Never new-up processors directly; resolve them from the injector.","In tests, inject a fake WebScraperService explicitly.","Fail fast at startup by touching all injected deps once."],"tags":["dependency-injection","initialization","web-search","testing"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}