{"record":{"id":"5b3695ca45679910","repo":"zylon-ai/private-gpt","slug":"web-fetching-is-not-properly-initialized-or-it-is","errorCode":null,"errorMessage":"Web fetching is not properly initialized or it is disabled. Consider enabling web fetching in settings.","messagePattern":"Web fetching is not properly initialized or it is disabled\\. Consider enabling web fetching in settings\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/web/web_scraper_service.py","lineNumber":80,"sourceCode":"    def is_initialized(self) -> bool:\n        return self._initialized\n\n    def _apply_ssl_environment(self) -> None:\n        ssl_config = self._settings.server.network.ssl\n        if ssl_config.cert_file:\n            os.environ[\"NODE_EXTRA_CA_CERTS\"] = ssl_config.cert_file\n            os.environ[\"SSL_CERT_FILE\"] = ssl_config.cert_file\n            logger.debug(f\"Custom CA certificate configured: {ssl_config.cert_file}\")\n\n        if ssl_config.cert_dir:\n            os.environ[\"SSL_CERT_DIR\"] = ssl_config.cert_dir\n            logger.debug(\n                f\"Custom CA certificate directory configured: {ssl_config.cert_dir}\"\n            )\n\n    async def _scrape_html(self, url: str) -> str:\n        if not self._initialized:\n            raise ValueError(\n                \"Web fetching is not properly initialized or it is disabled. \"\n                \"Consider enabling web fetching in settings.\"\n            )\n\n        provider = self._registry.get_provider(self._settings.web_fetch.provider)\n        _start_scrape = time.monotonic()\n        try:\n            content = await provider.scrape_html(\n                url, self._settings.web_fetch.timeout_seconds\n            )\n        except TimeoutError:\n            logger.warning(\n                f\"Timeout ({self._settings.web_fetch.timeout_seconds}s) scraping {url}\"\n            )\n            raise\n        except Exception as e:\n            logger.warning(f\"Error scraping {url}: {e}\")\n            raise","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/web/web_scraper_service.py#L62-L98","documentation":"ValueError raised by WebScraperService._scrape_html when the singleton was never marked initialized. _initialize() skips initialization when settings.web_fetch.enabled is False (the default), leaving _initialized False; every later scrape call trips this guard.","triggerScenarios":"Default settings (web_fetch.enabled defaults to false) combined with any code path that scrapes; enabling web search but not web fetch; settings profile/env var not loaded so the enabled flag stays default.","commonSituations":"Fresh deployment where web fetching was never turned on; users enabling `web_search.enabled` but forgetting `web_fetch.enabled`; YAML typo under a different profile so the override never applies.","solutions":["Set web_fetch.enabled: true in settings (profile yaml or env var PRIVATE_GPT_WEB_FETCH_ENABLED=true).","If web search is the goal, enable both web_fetch and web_search.","Verify with the service's is_initialized property after startup to catch config drift early."],"exampleFix":"# settings.yaml\nweb_fetch:\n  enabled: true  # was false / missing\n  provider: local","handlingStrategy":"validation","validationCode":"svc = get_global_injector().get(WebScraperService)\nif not svc.is_initialized:\n    raise ValueError('web_fetch.enabled is false — enable it before scraping')","typeGuard":"def scraper_ready(svc: WebScraperService) -> bool:\n    return svc.is_initialized","tryCatchPattern":"try:\n    html = await svc.scrape(url)\nexcept ValueError as e:\n    if 'Web fetching is not properly initialized' in str(e):\n        # config problem, not a page problem: surface to operator\n        raise RuntimeError('Fix settings: web_fetch.enabled=true') from e\n    raise","preventionTips":["Assert scraper.is_initialized during app startup, not at first scrape.","Keep web_fetch and web_search toggles together in config templates.","Add a config lint that warns when web_search.enabled=true but web_fetch.enabled=false."],"tags":["configuration","web-fetch","settings","initialization"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}