ScrapeGraphAI/Scrapegraph-ai · error · ValueError

No HTML body content found in the response.

Error message

No HTML body content found in the response.

What it means

In handle_web_source, after a successful HTTP 200 response, if response.text is empty or whitespace-only this ValueError is raised — the server returned a body-less page.

Source

Thrown at scrapegraphai/nodes/fetch_node.py:292

        Returns:
        dict: The updated state with the processed content.

        Raises:
        ValueError: If the fetched HTML content is empty or contains only whitespace.
        """

        self.logger.info(f"--- (Fetching HTML from: {source}) ---")
        if self.use_soup:
            # Apply configured timeout to blocking HTTP requests. If timeout is None,
            # don't pass the timeout argument (requests will block until completion).
            if self.timeout is None:
                response = requests.get(source)
            else:
                response = requests.get(source, timeout=self.timeout)
            if response.status_code == 200:
                if not response.text.strip():
                    raise ValueError("No HTML body content found in the response.")

                if not self.cut:
                    parsed_content = cleanup_html(response, source)

                if (
                    isinstance(self.llm_model, (ChatOpenAI, AzureChatOpenAI))
                    and not self.script_creator
                    or (self.force and not self.script_creator)
                ):
                    parsed_content = convert_to_md(source, parsed_content)

                compressed_document = [Document(page_content=parsed_content)]
            else:
                self.logger.warning(
                    f"Failed to retrieve contents from the webpage at url: {source}"
                )
        else:
            loader_kwargs = {}

View on GitHub (pinned to 532dfffbf6)

Solutions

1. Verify the URL returns HTML with curl -sL <url> | wc -c
2. Use the Chromium loader (browser_base or playwright-based fetching) for JS-heavy or protected sites
3. Retry with backoff for intermittent empty responses
4. Check for a wrong URL scheme/host

Example fix

# before
graph_config = {'source': 'https://example.com/api/empty'}
# after (render with browser)
graph_config = {'source': 'https://example.com', 'headless': True}

import requests
r = requests.get(url, timeout=30)
if r.status_code == 200 and not r.text.strip():
    # switch to browser-based fetching ahead of time
    graph_config['headless'] = True

for attempt in range(3):
    try:
        graph.run()
        break
    except ValueError as e:
        if 'No HTML body content found in the response' not in str(e) or attempt == 2:
            raise
        time.sleep(2 ** attempt)

When it happens

Trigger: Hitting an endpoint that returns 200 with an empty body (some APIs, redirects handled oddly, misconfigured servers); robots/anti-bot layer serving empty 200 responses.

Common situations: Scraping sites behind CDNAS/WAFs that return empty bodies to non-browser clients; URL pointing to a JSON API endpoint instead of an HTML page; intermittent server behavior.

Related errors

• Invalid input type: {input_type}
• PDF parsing exceeded timeout of {self.timeout} seconds
• pandas is not installed. Please install it using `pip instal
• No HTML body content found in the local source.
• The browserbase module is not installed.

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/f715b09f73bb3303. Report an issue: GitHub.