{"record":{"id":"817ca08920adda10","repo":"unclecode/crawl4ai","slug":"process-html-failed-to-extract-content-from-the-w","errorCode":null,"errorMessage":"Process HTML, Failed to extract content from the website: {url}","messagePattern":"Process HTML, Failed to extract content from the website: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_webcrawler.py","lineNumber":787,"sourceCode":"            scraping_strategy = config.scraping_strategy\n            if not scraping_strategy.logger:\n                scraping_strategy.logger = self.logger\n\n            # Process HTML content\n            params = config.__dict__.copy()\n            params.pop(\"url\", None)\n            # add keys from kwargs to params that doesn't exist in params\n            params.update({k: v for k, v in kwargs.items()\n                          if k not in params.keys()})\n\n            ################################\n            # Scraping Strategy Execution  #\n            ################################\n            result: ScrapingResult = scraping_strategy.scrap(\n                url, html, **params)\n\n            if result is None:\n                raise ValueError(\n                    f\"Process HTML, Failed to extract content from the website: {url}\"\n                )\n\n        except InvalidCSSSelectorError as e:\n            raise ValueError(str(e))\n        except Exception as e:\n            raise ValueError(\n                f\"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}\"\n            )\n\n        # Extract results - handle both dict and ScrapingResult\n        if isinstance(result, dict):\n            cleaned_html = sanitize_input_encode(\n                result.get(\"cleaned_html\", \"\"))\n            media = result.get(\"media\", {})\n            tables = media.pop(\"tables\", []) if isinstance(media, dict) else []\n            links = result.get(\"links\", {})\n            metadata = result.get(\"metadata\", {})","sourceCodeStart":769,"sourceCodeEnd":805,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_webcrawler.py#L769-L805","documentation":"In AsyncWebCrawler.aprocess_html, scraping_strategy.scrap(url, html, **params) returned None, which is treated as total extraction failure and raises this ValueError. The None result means the scraping pipeline produced no ScrapingResult at all (as opposed to an exception, which produces the sibling error with ': error: ...').","triggerScenarios":"Calling arun() with html= content that the content scraping strategy cannot process (empty body after sanitization, non-HTML input like pure JSON or binary text, or a page whose DOM matches nothing). Also custom ContentScrapingStrategy subclasses whose scrap() returns None instead of ScrapingResult.","commonSituations":"Using arun(html=...) to post-process saved HTML that was truncated or empty; custom scraping strategies not returning a result object; pages that are fully pruned by CSS selectors, leaving nothing to extract.","solutions":["Verify the html string you pass is non-empty, well-formed HTML (print len(html) and the first 200 chars).","If using css_selector or exclusion selectors, loosen them so the result is not entirely pruned.","If you implemented a custom scraping strategy, make scrap() always return a ScrapingResult (or dict) instead of None.","Reproduce with the live page: crawler.arun(url) instead of a stale cached html copy."],"exampleFix":"# before\nresult = await crawler.arun(url='', html=saved_html, config=config)  # saved_html may be ''\n\n# after\nif not saved_html or '<' not in saved_html:\n    raise ValueError('saved_html does not look like HTML')\nresult = await crawler.arun(url=url, html=saved_html, config=config)","handlingStrategy":"validation","validationCode":"def looks_like_html(html: str) -> bool:\n    return bool(html) and '<' in html and '</' in html","typeGuard":"def is_processable_html(html) -> bool:\n    return isinstance(html, str) and len(html.strip()) > 0 and '<' in html","tryCatchPattern":"try:\n    result = await crawler.arun(url, html=html, config=config)\nexcept ValueError as e:\n    if 'Failed to extract content' in str(e) and 'error:' not in str(e):\n        return None  # extraction yielded nothing; skip\n    raise","preventionTips":["Check html is non-empty and contains tags before calling arun(html=...).","Loosen over-aggressive css_selector/exclusion selectors.","Custom scraping strategies must always return ScrapingResult, never None."],"tags":["webcrawler","scraping","html-processing"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}