{"record":{"id":"a8bb51b9ad59635d","repo":"unclecode/crawl4ai","slug":"failed-to-extract-html-content-str-e","errorCode":null,"errorMessage":"Failed to extract HTML content: {str(e)}","messagePattern":"Failed to extract HTML content: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_crawler_strategy.py","lineNumber":1083,"sourceCode":"            elif config.css_selector:\n                try:\n                    selectors = [s.strip() for s in config.css_selector.split(',')]\n                    html_parts = []\n\n                    for selector in selectors:\n                        try:\n                            content = await self.adapter.evaluate(page,\n                                f\"\"\"Array.from(document.querySelectorAll(\"{selector}\"))\n                                    .map(el => el.outerHTML)\n                                    .join('')\"\"\"\n                            )\n                            html_parts.append(content)\n                        except Error as e:\n                            print(f\"Warning: Could not get content for selector '{selector}': {str(e)}\")\n\n                    html = f\"<div class='crawl4ai-result'>\\n\" + \"\\n\".join(html_parts) + \"\\n</div>\"\n                except Error as e:\n                    raise RuntimeError(f\"Failed to extract HTML content: {str(e)}\")\n            else:\n                html = await page.content()\n\n            await self.execute_hook(\n                \"before_return_html\", page=page, html=html, context=context, config=config\n            )\n\n            # Handle PDF, MHTML and screenshot generation\n            start_export_time = time.perf_counter()\n            pdf_data = None\n            screenshot_data = None\n            mhtml_data = None\n\n            if config.pdf:\n                pdf_data = await self.export_pdf(page)\n\n            if config.capture_mhtml:\n                mhtml_data = await self.capture_mhtml(page)","sourceCodeStart":1065,"sourceCodeEnd":1101,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_crawler_strategy.py#L1065-L1101","documentation":"Raised when config.css_selector is set and Playwright throws an Error while collecting the HTML for those selectors (beyond per-selector warnings which are only printed). The code builds a wrapper div from querySelectorAll(selector).outerHTML parts; if the overall evaluation fails with a Playwright Error it is wrapped in this RuntimeError.","triggerScenarios":"Setting CrawlerRunConfig(css_selector=...) with a syntactically invalid selector (e.g. unbalanced quotes or unsupported pseudo-classes like :has-text() which are Playwright-locator-only, not valid querySelectorAll syntax); or the page navigating away mid-evaluation causing an execution-context destruction Error.","commonSituations":"Copying Playwright locator syntax (:text(), :has-text()) into css_selector which must be plain CSS; selectors scraped from tooling that emit XPath ('//div') instead of CSS; races where js_code triggers navigation before extraction.","solutions":["Use a pure CSS selector valid for document.querySelectorAll (no Playwright pseudo-classes, no XPath).","Test the selector with document.querySelectorAll('<sel>') in devtools first.","If js_code causes navigation, wait for the new page before extraction (set wait_for or delay_before_return_html).","Catch RuntimeError per page during batch crawls and log the offending selector."],"exampleFix":"// before\ncfg = CrawlerRunConfig(css_selector=\"div:has-text('price')\")\n\n// after\ncfg = CrawlerRunConfig(css_selector=\"div.price\")","handlingStrategy":"validation","validationCode":"import re\n\n_CSS_CHECK = re.compile(r\"^[.#]?[\\w\\->,: .()\\[\\]='\\\"]+$\")\n\ndef is_plain_css(selector: str) -> bool:\n    return bool(selector) and \":has-text\" not in selector and \":text\" not in selector and not selector.startswith((\"//\", \"xpath=\"))","typeGuard":null,"tryCatchPattern":"try:\n    result = await crawler.arun(url, config=cfg)\nexcept RuntimeError as e:\n    if \"Failed to extract HTML content\" in str(e):\n        cfg.css_selector = None  # fall back to full-page content\n        result = await crawler.arun(url, config=cfg)","preventionTips":["Use only querySelectorAll-valid CSS","No Playwright pseudo-classes or XPath in css_selector","Avoid js_code that navigates during extraction"],"tags":["css-selector","extraction","playwright"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}