What causes "Failed to extract HTML content: {str(e)}" in unclecode/crawl4ai?

Use a pure CSS selector valid for document.querySelectorAll (no Playwright pseudo-classes, no XPath).

unclecode/crawl4ai · error · RuntimeError

Failed to extract HTML content: {str(e)}

Error message

Failed to extract HTML content: {str(e)}

What it means

Raised when config.css_selector is set and Playwright throws an Error while collecting the HTML for those selectors (beyond per-selector warnings which are only printed). The code builds a wrapper div from querySelectorAll(selector).outerHTML parts; if the overall evaluation fails with a Playwright Error it is wrapped in this RuntimeError.

Source

Thrown at crawl4ai/async_crawler_strategy.py:1083

            elif config.css_selector:
                try:
                    selectors = [s.strip() for s in config.css_selector.split(',')]
                    html_parts = []

                    for selector in selectors:
                        try:
                            content = await self.adapter.evaluate(page,
                                f"""Array.from(document.querySelectorAll("{selector}"))
                                    .map(el => el.outerHTML)
                                    .join('')"""
                            )
                            html_parts.append(content)
                        except Error as e:
                            print(f"Warning: Could not get content for selector '{selector}': {str(e)}")

                    html = f"<div class='crawl4ai-result'>\n" + "\n".join(html_parts) + "\n</div>"
                except Error as e:
                    raise RuntimeError(f"Failed to extract HTML content: {str(e)}")
            else:
                html = await page.content()

            await self.execute_hook(
                "before_return_html", page=page, html=html, context=context, config=config
            )

            # Handle PDF, MHTML and screenshot generation
            start_export_time = time.perf_counter()
            pdf_data = None
            screenshot_data = None
            mhtml_data = None

            if config.pdf:
                pdf_data = await self.export_pdf(page)

            if config.capture_mhtml:
                mhtml_data = await self.capture_mhtml(page)

View on GitHub (pinned to 7e80152142)

Solutions

Use a pure CSS selector valid for document.querySelectorAll (no Playwright pseudo-classes, no XPath).
Test the selector with document.querySelectorAll('<sel>') in devtools first.
If js_code causes navigation, wait for the new page before extraction (set wait_for or delay_before_return_html).
Catch RuntimeError per page during batch crawls and log the offending selector.

Example fix

// before
cfg = CrawlerRunConfig(css_selector="div:has-text('price')")

// after
cfg = CrawlerRunConfig(css_selector="div.price")

Defensive patterns

Strategy: validation

Validate before calling

import re

_CSS_CHECK = re.compile(r"^[.#]?[\w\->,: .()\[\]='\"]+$")

def is_plain_css(selector: str) -> bool:
    return bool(selector) and ":has-text" not in selector and ":text" not in selector and not selector.startswith(("//", "xpath="))

Try / catch

try:
    result = await crawler.arun(url, config=cfg)
except RuntimeError as e:
    if "Failed to extract HTML content" in str(e):
        cfg.css_selector = None  # fall back to full-page content
        result = await crawler.arun(url, config=cfg)

Prevention

Use only querySelectorAll-valid CSS
No Playwright pseudo-classes or XPath in css_selector
Avoid js_code that navigates during extraction

When it happens

Trigger: Setting CrawlerRunConfig(css_selector=...) with a syntactically invalid selector (e.g. unbalanced quotes or unsupported pseudo-classes like :has-text() which are Playwright-locator-only, not valid querySelectorAll syntax); or the page navigating away mid-evaluation causing an execution-context destruction Error.

Common situations: Copying Playwright locator syntax (:text(), :has-text()) into css_selector which must be plain CSS; selectors scraped from tooling that emit XPath ('//div') instead of CSS; races where js_code triggers navigation before extraction.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/a8bb51b9ad59635d. Report an issue: GitHub.