unclecode/crawl4ai · error · ValueError

{e}

Error message

{e}

What it means

ValueError re-raise from AsyncWebCrawler.process_html: an InvalidCSSSelectorError escaped the scraping strategy and is re-raised verbatim (str(e)) - no URL is attached, so the message is exactly the selector error text.

Source

Thrown at crawl4ai/async_webcrawler.py:789:1850

            params = config.__dict__.copy()
            params.pop("url", None)
            # add keys from kwargs to params that doesn't exist in params
            params.update({k: v for k, v in kwargs.items()
                          if k not in params.keys()})

            ################################
            # Scraping Strategy Execution  #
            ################################
            result: ScrapingResult = scraping_strategy.scrap(
                url, html, **params)

            if result is None:
                raise ValueError(
                    f"Process HTML, Failed to extract content from the website: {url}"
                )

        except InvalidCSSSelectorError as e:
            raise ValueError(str(e))
        except Exception as e:
            raise ValueError(
                f"Process HTML, Failed to extract content from the website: {url}, error: {str(e)}"
            )

        # Extract results - handle both dict and ScrapingResult
        if isinstance(result, dict):
            cleaned_html = sanitize_input_encode(
                result.get("cleaned_html", ""))
            media = result.get("media", {})
            links = result.get("links", {})
            metadata = result.get("metadata", {})
        else:
            cleaned_html = sanitize_input_encode(result.cleaned_html)
            media = result.media.model_dump()
            links = result.links.model_dump()
            metadata = result.metadata

View on GitHub (pinned to 7e80152142)

Solutions

1. Test each selector locally: BeautifulSoup('<div></div>', 'html.parser').select(your_selector) must not raise
2. Quote attribute values and balance brackets: a[href^="https://"] not a[href^=https://]
3. If you need Playwright-specific selectors, pass them via js_code/wait_for rather than css_selector
4. Wrap arun in try/except ValueError and log which URL/selector combo failed when batch-crawling

Example fix

# before
config = CrawlerRunConfig(css_selector='article >> .content')

# after
config = CrawlerRunConfig(css_selector='article .content')

from bs4 import BeautifulSoup

def selectors_valid(*selectors) -> bool:
    probe = BeautifulSoup("<div><a href='#'>x</a></div>", "html.parser")
    try:
        for s in selectors:
            if s:
                probe.select(s)
        return True
    except Exception:
        return False

def is_valid_css_selector(s) -> bool:
    if not s:
        return True
    try:
        BeautifulSoup("<div></div>", "html.parser").select(s)
        return True
    except Exception:
        return False

try:
    await crawler.arun(url=url, config=config)
except ValueError as e:
    msg = str(e)
    if "selector" in msg.lower() or "invalid" in msg.lower():
        config = CrawlerRunConfig(**{**vars_like(config), "css_selector": fix_selector(config.css_selector)})
    else:
        raise

When it happens

Trigger: config.css_selector, content_selector, or exclude_css contains an invalid CSS selector such as 'a[href', '>>div', or a stray pseudo-class the parser rejects; the strategy validates selectors and throws InvalidCSSSelectorError during scrap().

Common situations: Hand-written selectors with typos; selectors generated from templates containing empty fragments; using Playwright/JS-style selectors (':has-text(...)') where a CSS engine is used; changing exclude_css values between runs without re-validating.

Related errors

• Invalid CSS selector , No elements found for CSS selector: {
• Invalid CSS selector: {css_selector}
• Invalid CSS selector, No elements found for CSS selector: {c
• Hook '{hook_name}' must be a callable function, got {type(ho
• URL must have a valid hostname

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/aa2a25b6943b976a. Report an issue: GitHub.