{"record":{"id":"269355982f5b0b70","repo":"zylon-ai/private-gpt","slug":"can-t-extract-information-from-the-provided-url-a","errorCode":null,"errorMessage":"Can't extract information from the provided url, automated tools do not work on this page.","messagePattern":"Can't extract information from the provided url, automated tools do not work on this page\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"warning","filePath":"private_gpt/components/web/web_scraper_service.py","lineNumber":121,"sourceCode":"        )\n        return content\n\n    async def scrape(self, url: str) -> WebScraperResult:\n        _start = time.monotonic()\n        logger.debug(f\"Scrape start: {url}\")\n        result = WebScraperResult()\n        result.url = url\n        result.html_content = await self._scrape_html(url)\n\n        result.markdown_content = await asyncio.to_thread(\n            self._html_to_markdown,\n            result.html_content,\n        )\n\n        if not result.markdown_content.strip():\n            _elapsed = time.monotonic() - _start\n            logger.warning(f\"Cannot extract text from {url} ({_elapsed:.2f}s)\")\n            raise Exception(\n                \"Can't extract information from the provided url, \"\n                \"automated tools do not work on this page.\"\n            )\n        _elapsed = time.monotonic() - _start\n        logger.debug(\n            f\"Scrape complete: {url} ({_elapsed:.2f}s, \"\n            f\"html={len(result.html_content or '')}, \"\n            f\"md={len(result.markdown_content or '')})\"\n        )\n        return result\n\n    async def scrape_max_compress(self, url: str) -> WebScraperResult:\n        _start = time.monotonic()\n        logger.debug(f\"Max-compress scrape start: {url}\")\n        result: WebScraperResult = WebScraperResult()\n        result.url = url\n\n        result.html_content = await self._scrape_html(url)","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/web/web_scraper_service.py#L103-L139","documentation":"Generic Exception raised by WebScraperService.scrape when the page HTML was fetched successfully but produced an empty (or whitespace-only) markdown after conversion. It signals the content-extraction stage lost: either the page relies on JavaScript rendering the scraper did not execute, or the HTML cleaner stripped everything (boilerplate-only markup, framesets, binary/non-HTML responses parsed as HTML).","triggerScenarios":"Scraping an SPA whose DOM is empty on first paint; a URL that returns a redirect page, consent wall, or CAPTCHA; a PDF/image served with an HTML content type wrapper; a page whose entire body is <script> tags that the cleaner removes; anti-bot pages served to non-browser clients.","commonSituations":"React/Vue/Angular sites scraped with a plain HTTP provider; EU consent walls; Cloudflare challenges; endpoints that need cookies/auth headers the scraper does not send.","solutions":["Open the URL with JavaScript disabled — if it is blank, switch web_fetch.provider to a browser-based one (local/opensandbox Playwright) or wait for full render.","Increase web_fetch.timeout_seconds so lazy-loaded content lands before extraction.","Check for consent walls/CAPTCHAs and pass required cookies or a different URL.","Handle this exception per-URL (as scraped_content_processor does with gather(return_exceptions=True)) so one dead page does not kill a batch."],"exampleFix":"# before\nresults = await asyncio.gather(*[svc.scrape(u) for u in urls])\n# one empty page raises and fails everything\n\n# after\nresults = await asyncio.gather(\n    *[svc.scrape(u) for u in urls], return_exceptions=True\n)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"def is_unextractable_page(exc: BaseException) -> bool:\n    return isinstance(exc, Exception) and 'automated tools do not work' in str(exc)","tryCatchPattern":"try:\n    result = await svc.scrape(url)\nexcept Exception as e:\n    if 'automated tools do not work' in str(e):\n        logger.warning('unscrapable page, skipping: %s', url)\n        continue  # or fall back to raw HTML / search snippet\\n    raise","preventionTips":["Always scrape with gather(..., return_exceptions=True) in batch flows.","Prefer a browser-based web_fetch provider for JS-heavy domains.","Maintain a denylist of known-unscrapable hosts (consent walls, SPAs) instead of retrying them."],"tags":["scraping","content-extraction","spa","anti-bot"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}