{"record":{"id":"f715b09f73bb3303","repo":"ScrapeGraphAI/Scrapegraph-ai","slug":"no-html-body-content-found-in-the-response","errorCode":null,"errorMessage":"No HTML body content found in the response.","messagePattern":"No HTML body content found in the response\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scrapegraphai/nodes/fetch_node.py","lineNumber":292,"sourceCode":"\n        Returns:\n        dict: The updated state with the processed content.\n\n        Raises:\n        ValueError: If the fetched HTML content is empty or contains only whitespace.\n        \"\"\"\n\n        self.logger.info(f\"--- (Fetching HTML from: {source}) ---\")\n        if self.use_soup:\n            # Apply configured timeout to blocking HTTP requests. If timeout is None,\n            # don't pass the timeout argument (requests will block until completion).\n            if self.timeout is None:\n                response = requests.get(source)\n            else:\n                response = requests.get(source, timeout=self.timeout)\n            if response.status_code == 200:\n                if not response.text.strip():\n                    raise ValueError(\"No HTML body content found in the response.\")\n\n                if not self.cut:\n                    parsed_content = cleanup_html(response, source)\n\n                if (\n                    isinstance(self.llm_model, (ChatOpenAI, AzureChatOpenAI))\n                    and not self.script_creator\n                    or (self.force and not self.script_creator)\n                ):\n                    parsed_content = convert_to_md(source, parsed_content)\n\n                compressed_document = [Document(page_content=parsed_content)]\n            else:\n                self.logger.warning(\n                    f\"Failed to retrieve contents from the webpage at url: {source}\"\n                )\n        else:\n            loader_kwargs = {}","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/532dfffbf6ee823a6c9cf8cfedc24a93bf026780/scrapegraphai/nodes/fetch_node.py#L274-L310","documentation":"In handle_web_source, after a successful HTTP 200 response, if response.text is empty or whitespace-only this ValueError is raised — the server returned a body-less page.","triggerScenarios":"Hitting an endpoint that returns 200 with an empty body (some APIs, redirects handled oddly, misconfigured servers); robots/anti-bot layer serving empty 200 responses.","commonSituations":"Scraping sites behind CDNAS/WAFs that return empty bodies to non-browser clients; URL pointing to a JSON API endpoint instead of an HTML page; intermittent server behavior.","solutions":["Verify the URL returns HTML with curl -sL <url> | wc -c","Use the Chromium loader (browser_base or playwright-based fetching) for JS-heavy or protected sites","Retry with backoff for intermittent empty responses","Check for a wrong URL scheme/host"],"exampleFix":"# before\ngraph_config = {'source': 'https://example.com/api/empty'}\n# after (render with browser)\ngraph_config = {'source': 'https://example.com', 'headless': True}","handlingStrategy":"retry","validationCode":"import requests\nr = requests.get(url, timeout=30)\nif r.status_code == 200 and not r.text.strip():\n    # switch to browser-based fetching ahead of time\n    graph_config['headless'] = True","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        graph.run()\n        break\n    except ValueError as e:\n        if 'No HTML body content found in the response' not in str(e) or attempt == 2:\n            raise\n        time.sleep(2 ** attempt)","preventionTips":["Pre-check URLs with a HEAD/GET probe in orchestration code","Use Chromium/BrowserBase fetching for JS-heavy or protected sites"],"tags":["scrapegraphai","fetch-node","web-scraping","empty-response"],"backgroundTag":"empty-http-response","analyzedSha":"532dfffbf6ee823a6c9cf8cfedc24a93bf026780","analyzedAt":"2026-08-28T15:19:38.821Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}