zylon-ai/private-gpt · warning · Exception

Can't extract information from the provided url, automated t

Error message

Can't extract information from the provided url, automated tools do not work on this page.

What it means

Generic Exception raised by WebScraperService.scrape when the page HTML was fetched successfully but produced an empty (or whitespace-only) markdown after conversion. It signals the content-extraction stage lost: either the page relies on JavaScript rendering the scraper did not execute, or the HTML cleaner stripped everything (boilerplate-only markup, framesets, binary/non-HTML responses parsed as HTML).

Source

Thrown at private_gpt/components/web/web_scraper_service.py:121

        )
        return content

    async def scrape(self, url: str) -> WebScraperResult:
        _start = time.monotonic()
        logger.debug(f"Scrape start: {url}")
        result = WebScraperResult()
        result.url = url
        result.html_content = await self._scrape_html(url)

        result.markdown_content = await asyncio.to_thread(
            self._html_to_markdown,
            result.html_content,
        )

        if not result.markdown_content.strip():
            _elapsed = time.monotonic() - _start
            logger.warning(f"Cannot extract text from {url} ({_elapsed:.2f}s)")
            raise Exception(
                "Can't extract information from the provided url, "
                "automated tools do not work on this page."
            )
        _elapsed = time.monotonic() - _start
        logger.debug(
            f"Scrape complete: {url} ({_elapsed:.2f}s, "
            f"html={len(result.html_content or '')}, "
            f"md={len(result.markdown_content or '')})"
        )
        return result

    async def scrape_max_compress(self, url: str) -> WebScraperResult:
        _start = time.monotonic()
        logger.debug(f"Max-compress scrape start: {url}")
        result: WebScraperResult = WebScraperResult()
        result.url = url

        result.html_content = await self._scrape_html(url)

View on GitHub (pinned to 4a030776a3)

Solutions

1. Open the URL with JavaScript disabled — if it is blank, switch web_fetch.provider to a browser-based one (local/opensandbox Playwright) or wait for full render.
2. Increase web_fetch.timeout_seconds so lazy-loaded content lands before extraction.
3. Check for consent walls/CAPTCHAs and pass required cookies or a different URL.
4. Handle this exception per-URL (as scraped_content_processor does with gather(return_exceptions=True)) so one dead page does not kill a batch.

Example fix

# before
results = await asyncio.gather(*[svc.scrape(u) for u in urls])
# one empty page raises and fails everything

# after
results = await asyncio.gather(
    *[svc.scrape(u) for u in urls], return_exceptions=True
)

def is_unextractable_page(exc: BaseException) -> bool:
    return isinstance(exc, Exception) and 'automated tools do not work' in str(exc)

try:
    result = await svc.scrape(url)
except Exception as e:
    if 'automated tools do not work' in str(e):
        logger.warning('unscrapable page, skipping: %s', url)
        continue  # or fall back to raw HTML / search snippet\n    raise

When it happens

Trigger: Scraping an SPA whose DOM is empty on first paint; a URL that returns a redirect page, consent wall, or CAPTCHA; a PDF/image served with an HTML content type wrapper; a page whose entire body is <script> tags that the cleaner removes; anti-bot pages served to non-browser clients.

Common situations: React/Vue/Angular sites scraped with a plain HTTP provider; EU consent walls; Cloudflare challenges; endpoints that need cookies/auth headers the scraper does not send.

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/269355982f5b0b70. Report an issue: GitHub.