{"record":{"id":"9dcdfeb419fcb685","repo":"unclecode/crawl4ai","slug":"process-html-failed-to-extract-content-from-the-w-9dcdfe","errorCode":null,"errorMessage":"Process HTML, Failed to extract content from the website: {url}","messagePattern":"Process HTML, Failed to extract content from the website: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_webcrawler.py:787","lineNumber":1846,"sourceCode":"            if not scraping_strategy.logger:\n                scraping_strategy.logger = self.logger\n\n            # Process HTML content\n            params = config.__dict__.copy()\n            params.pop(\"url\", None)\n            # add keys from kwargs to params that doesn't exist in params\n            params.update({k: v for k, v in kwargs.items()\n                          if k not in params.keys()})\n\n            ################################\n            # Scraping Strategy Execution  #\n            ################################\n            result: ScrapingResult = scraping_strategy.scrap(\n                url, html, **params)\n\n            if result is None:\n                raise ValueError(\n                    f\"Process HTML, Failed to extract content from the website: {url}\"\n                )\n\n        except InvalidCSSSelectorError as e:\n            raise ValueError(str(e))\n        except Exception as e:\n            raise ValueError(\n                f\"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}\"\n            )\n\n        # Extract results - handle both dict and ScrapingResult\n        if isinstance(result, dict):\n            cleaned_html = sanitize_input_encode(\n                result.get(\"cleaned_html\", \"\"))\n            media = result.get(\"media\", {})\n            links = result.get(\"links\", {})\n            metadata = result.get(\"metadata\", {})\n        else:\n            cleaned_html = sanitize_input_encode(result.cleaned_html)","sourceCodeStart":1828,"sourceCodeEnd":1864,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/deploy/docker/c4ai-code-context.md#L1828-L1864","documentation":"ValueError from AsyncWebCrawler.process_html: the scraping strategy's scrap(url, html, **params) returned None, meaning content extraction produced no result object at all (distinct from an exception, which takes the other branch and appends ', error: ...').","triggerScenarios":"A scraping strategy (built-in WebScrapingStrategy or a custom one) that returns None on unhandled input - empty html string, malformed document the strategy gives up on, or a custom strategy whose code path forgets to return.","commonSituations":"Custom ScrapingStrategy subclass that returns None in an edge-case branch; passing pre-fetched html that is an empty string or a non-HTML payload (JSON error page, binary); version mismatch where the strategy's expected params differ from what process_html forwards.","solutions":["If html comes from your own fetch, assert it is non-empty and looks like HTML before calling arun(html=...)","In a custom strategy, guarantee a ScrapingResult is always returned; raise instead of returning None so the real cause surfaces","Log the html length and first 200 chars when the error fires to confirm what the strategy saw","Reproduce with the built-in WebScrapingStrategy to decide whether the bug is in your strategy or the input"],"exampleFix":"# before (custom strategy)\ndef scrap(self, url, html, **kwargs):\n    if not html:\n        return None  # -> ValueError upstream\n\n# after\ndef scrap(self, url, html, **kwargs):\n    if not html:\n        raise ValueError(f\"empty html for {url}\")\n    return ScrapingResult(cleaned_html=html, media={}, links={}, metadata={})","handlingStrategy":"type-guard","validationCode":"assert isinstance(html, str) and html.strip() and '<' in html, \"html must be a non-empty, HTML-like string\"","typeGuard":"def is_processable_html(h) -> bool:\n    return isinstance(h, str) and len(h.strip()) > 0 and '<' in h","tryCatchPattern":"try:\n    result = await crawler.arun(url=url, html=html, config=config)\nexcept ValueError as e:\n    if \"Failed to extract content\" in str(e) and \"error:\" not in str(e):\n        logger.error(\"strategy returned None for %s (len=%d)\", url, len(html or ''))\n        result = None","preventionTips":["Custom ScrapingStrategy implementations must return a ScrapingResult on every path - raise on failure instead of returning None","Log html size and a preview before processing external HTML payloads"],"tags":["crawl4ai","extraction","scraping","crawler"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}