unclecode/crawl4ai · error · ValueError

Process HTML, Failed to extract content from the website: {u

Error message

Process HTML, Failed to extract content from the website: {url}

What it means

In AsyncWebCrawler.aprocess_html, scraping_strategy.scrap(url, html, **params) returned None, which is treated as total extraction failure and raises this ValueError. The None result means the scraping pipeline produced no ScrapingResult at all (as opposed to an exception, which produces the sibling error with ': error: ...').

Source

Thrown at crawl4ai/async_webcrawler.py:787

            scraping_strategy = config.scraping_strategy
            if not scraping_strategy.logger:
                scraping_strategy.logger = self.logger

            # Process HTML content
            params = config.__dict__.copy()
            params.pop("url", None)
            # add keys from kwargs to params that doesn't exist in params
            params.update({k: v for k, v in kwargs.items()
                          if k not in params.keys()})

            ################################
            # Scraping Strategy Execution  #
            ################################
            result: ScrapingResult = scraping_strategy.scrap(
                url, html, **params)

            if result is None:
                raise ValueError(
                    f"Process HTML, Failed to extract content from the website: {url}"
                )

        except InvalidCSSSelectorError as e:
            raise ValueError(str(e))
        except Exception as e:
            raise ValueError(
                f"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}"
            )

        # Extract results - handle both dict and ScrapingResult
        if isinstance(result, dict):
            cleaned_html = sanitize_input_encode(
                result.get("cleaned_html", ""))
            media = result.get("media", {})
            tables = media.pop("tables", []) if isinstance(media, dict) else []
            links = result.get("links", {})
            metadata = result.get("metadata", {})

View on GitHub (pinned to 7e80152142)

Solutions

Verify the html string you pass is non-empty, well-formed HTML (print len(html) and the first 200 chars).
If using css_selector or exclusion selectors, loosen them so the result is not entirely pruned.
If you implemented a custom scraping strategy, make scrap() always return a ScrapingResult (or dict) instead of None.
Reproduce with the live page: crawler.arun(url) instead of a stale cached html copy.

Example fix

# before
result = await crawler.arun(url='', html=saved_html, config=config)  # saved_html may be ''

# after
if not saved_html or '<' not in saved_html:
    raise ValueError('saved_html does not look like HTML')
result = await crawler.arun(url=url, html=saved_html, config=config)

Defensive patterns

Strategy: validation

Validate before calling

def looks_like_html(html: str) -> bool:
    return bool(html) and '<' in html and '</' in html

Type guard

def is_processable_html(html) -> bool:
    return isinstance(html, str) and len(html.strip()) > 0 and '<' in html

Try / catch

try:
    result = await crawler.arun(url, html=html, config=config)
except ValueError as e:
    if 'Failed to extract content' in str(e) and 'error:' not in str(e):
        return None  # extraction yielded nothing; skip
    raise

Prevention

Check html is non-empty and contains tags before calling arun(html=...).
Loosen over-aggressive css_selector/exclusion selectors.
Custom scraping strategies must always return ScrapingResult, never None.

When it happens

Trigger: Calling arun() with html= content that the content scraping strategy cannot process (empty body after sanitization, non-HTML input like pure JSON or binary text, or a page whose DOM matches nothing). Also custom ContentScrapingStrategy subclasses whose scrap() returns None instead of ScrapingResult.

Common situations: Using arun(html=...) to post-process saved HTML that was truncated or empty; custom scraping strategies not returning a result object; pages that are fully pruned by CSS selectors, leaving nothing to extract.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/817ca08920adda10. Report an issue: GitHub.