{"record":{"id":"4e7fd69b6f45fe12","repo":"unclecode/crawl4ai","slug":"process-html-failed-to-extract-content-from-the-w-4e7fd6","errorCode":null,"errorMessage":"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}","messagePattern":"Process HTML, Failed to extract content from the website: (.+?), error: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_webcrawler.py","lineNumber":794,"sourceCode":"            # add keys from kwargs to params that doesn't exist in params\n            params.update({k: v for k, v in kwargs.items()\n                          if k not in params.keys()})\n\n            ################################\n            # Scraping Strategy Execution  #\n            ################################\n            result: ScrapingResult = scraping_strategy.scrap(\n                url, html, **params)\n\n            if result is None:\n                raise ValueError(\n                    f\"Process HTML, Failed to extract content from the website: {url}\"\n                )\n\n        except InvalidCSSSelectorError as e:\n            raise ValueError(str(e))\n        except Exception as e:\n            raise ValueError(\n                f\"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}\"\n            )\n\n        # Extract results - handle both dict and ScrapingResult\n        if isinstance(result, dict):\n            cleaned_html = sanitize_input_encode(\n                result.get(\"cleaned_html\", \"\"))\n            media = result.get(\"media\", {})\n            tables = media.pop(\"tables\", []) if isinstance(media, dict) else []\n            links = result.get(\"links\", {})\n            metadata = result.get(\"metadata\", {})\n        else:\n            cleaned_html = sanitize_input_encode(result.cleaned_html)\n            # media = result.media.model_dump()\n            # tables = media.pop(\"tables\", [])\n            # links = result.links.model_dump()\n            media = result.media.model_dump() if hasattr(result.media, 'model_dump') else result.media\n            tables = media.pop(\"tables\", []) if isinstance(media, dict) else []","sourceCodeStart":776,"sourceCodeEnd":812,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_webcrawler.py#L776-L812","documentation":"The catch-all handler in aprocess_html: any exception raised inside the scraping/processing block (other than InvalidCSSSelectorError, which is re-raised verbatim) is wrapped in this ValueError with the original message appended. The root cause is in {str(e)}; this wrapper only adds the failing URL context.","triggerScenarios":"Exceptions thrown by the content scraping pipeline: malformed input HTML crashing BeautifulSoup/lxml, broken extraction strategies (LLM API errors inside an ExtractionStrategy invoked via params), errors in html2text conversion, or bugs in user-supplied scrap hooks. InvalidCSSSelectorError from bad css_selector is raised separately as its own message.","commonSituations":"Passing invalid CSS selectors (those surface as InvalidCSSSelectorError text); corrupted or truncated saved HTML; extraction strategies requiring API keys that are missing; version mismatches in html2text/lxml after upgrades.","solutions":["Read the trailing 'error: ...' part of the message — it contains the true exception; fix that root cause.","For InvalidCSSSelectorError text, fix the css_selector syntax in CrawlerRunConfig (e.g. unbalanced quotes, unsupported pseudo-selectors).","Validate/re-serialize the input HTML before passing html= (parse it with BeautifulSoup and re-encode).","If an extraction strategy fails, check its config/API keys or set config.extraction_strategy to None to isolate."],"exampleFix":"# before\nresult = await crawler.arun(url, html=html, config=CrawlerRunConfig(css_selector=\"a[href='\"))\n\n# after\nresult = await crawler.arun(url, html=html, config=CrawlerRunConfig(css_selector=\"a[href]\"))","handlingStrategy":"try-catch","validationCode":"from bs4 import BeautifulSoup\ndef is_parseable_html(html: str) -> bool:\n    try:\n        BeautifulSoup(html, 'lxml')\n        return True\n    except Exception:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    result = await crawler.arun(url, html=html, config=config)\nexcept ValueError as e:\n    msg = str(e)\n    if 'error:' in msg:\n        root = msg.split('error:', 1)[1].strip()\n        logger.warning('scrape failed for %s: %s', url, root)\n    else:\n        raise","preventionTips":["Log the 'error:' suffix — it holds the root cause, the wrapper does not.","Test css_selector values with a CSS parser before use.","Keep extraction strategies isolated so you can disable them to bisect failures."],"tags":["webcrawler","scraping","wrapper-exception"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}