unclecode/crawl4ai · error · ValueError

Process HTML, Failed to extract content from the website: {u

Error message

Process HTML, Failed to extract content from the website: {url}

What it means

ValueError from AsyncWebCrawler.process_html: the scraping strategy's scrap(url, html, **params) returned None, meaning content extraction produced no result object at all (distinct from an exception, which takes the other branch and appends ', error: ...').

Source

Thrown at crawl4ai/async_webcrawler.py:787:1846

            if not scraping_strategy.logger:
                scraping_strategy.logger = self.logger

            # Process HTML content
            params = config.__dict__.copy()
            params.pop("url", None)
            # add keys from kwargs to params that doesn't exist in params
            params.update({k: v for k, v in kwargs.items()
                          if k not in params.keys()})

            ################################
            # Scraping Strategy Execution  #
            ################################
            result: ScrapingResult = scraping_strategy.scrap(
                url, html, **params)

            if result is None:
                raise ValueError(
                    f"Process HTML, Failed to extract content from the website: {url}"
                )

        except InvalidCSSSelectorError as e:
            raise ValueError(str(e))
        except Exception as e:
            raise ValueError(
                f"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}"
            )

        # Extract results - handle both dict and ScrapingResult
        if isinstance(result, dict):
            cleaned_html = sanitize_input_encode(
                result.get("cleaned_html", ""))
            media = result.get("media", {})
            links = result.get("links", {})
            metadata = result.get("metadata", {})
        else:
            cleaned_html = sanitize_input_encode(result.cleaned_html)

View on GitHub (pinned to 7e80152142)

Solutions

1. If html comes from your own fetch, assert it is non-empty and looks like HTML before calling arun(html=...)
2. In a custom strategy, guarantee a ScrapingResult is always returned; raise instead of returning None so the real cause surfaces
3. Log the html length and first 200 chars when the error fires to confirm what the strategy saw
4. Reproduce with the built-in WebScrapingStrategy to decide whether the bug is in your strategy or the input

Example fix

# before (custom strategy)
def scrap(self, url, html, **kwargs):
    if not html:
        return None  # -> ValueError upstream

# after
def scrap(self, url, html, **kwargs):
    if not html:
        raise ValueError(f"empty html for {url}")
    return ScrapingResult(cleaned_html=html, media={}, links={}, metadata={})

assert isinstance(html, str) and html.strip() and '<' in html, "html must be a non-empty, HTML-like string"

def is_processable_html(h) -> bool:
    return isinstance(h, str) and len(h.strip()) > 0 and '<' in h

try:
    result = await crawler.arun(url=url, html=html, config=config)
except ValueError as e:
    if "Failed to extract content" in str(e) and "error:" not in str(e):
        logger.error("strategy returned None for %s (len=%d)", url, len(html or ''))
        result = None

When it happens

Trigger: A scraping strategy (built-in WebScrapingStrategy or a custom one) that returns None on unhandled input - empty html string, malformed document the strategy gives up on, or a custom strategy whose code path forgets to return.

Common situations: Custom ScrapingStrategy subclass that returns None in an edge-case branch; passing pre-fetched html that is an empty string or a non-HTML payload (JSON error page, binary); version mismatch where the strategy's expected params differ from what process_html forwards.

Related errors

• Process HTML, Failed to extract content from the website: {u
• extraction_strategy must be an instance of ExtractionStrateg
• Invalid URL, make sure the URL is a non-empty string
• {e}
• extraction_strategy must be an instance of ExtractionStrateg

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/9dcdfeb419fcb685. Report an issue: GitHub.