unclecode/crawl4ai · error · InvalidCSSSelectorError

Invalid CSS selector , No elements found for CSS selector

Error message

Invalid CSS selector , No elements found for CSS selector: {css_selector}

What it means

Raised by get_content_of_website (crawl4ai/utils.py:925) as InvalidCSSSelectorError when a css_selector was supplied to the HTML extraction helper but BeautifulSoup's body.select(css_selector) returned zero elements. Note the message is slightly misleading: the selector may be syntactically valid; it simply matched nothing in the parsed <body> (or the page has no <body>, making body None and the select fail). It exists so callers learn their extraction selector did not match the fetched HTML.

Solutions

  1. Verify the selector against the exact HTML string being passed (open it or print soup.body.prettify()) — the DOM you select on is the static parsed body, not the rendered page.
  2. Loosen the selector ("article" instead of "div.article-content.prose") or provide a fallback list of selectors tried in order.
  3. For JS-rendered content, fetch via AsyncPlaywrightCrawlerStrategy / the main AsyncWebcrawler with js_code or wait_for, then pass the rendered HTML (or use CrawlerRunConfig.css_selector which handles this in the pipeline).
  4. Validate the selector syntax first with soup.select on the soup object and catch InvalidCSSSelectorError to degrade gracefully.

Example fix

// before
result = get_content_of_website(url, html, css_selector="main.post-content")

// after
from crawl4ai.utils import InvalidCSSSelectorError
for sel in ("main.post-content", "article", "body"):
    try:
        result = get_content_of_website(url, html, css_selector=sel)
        break
    except InvalidCSSSelectorError:
        continue
Defensive patterns

Strategy: validation

Validate before calling

from bs4 import BeautifulSoup

def selector_matches(html: str, css_selector: str) -> bool:
    soup = BeautifulSoup(html, "html.parser")
    body = soup.body or soup
    try:
        return bool(body.select(css_selector))
    except Exception:
        return False

Try / catch

from crawl4ai.utils import InvalidCSSSelectorError
try:
    content = get_content_of_website(url, html, css_selector=sel)
except InvalidCSSSelectorError as e:
    logger.warning("selector %r matched nothing: %s", sel, e)
    content = get_content_of_website(url, html)  # fallback: full body

Prevention

When it happens

Trigger: Calling get_content_of_website(url, html, css_selector="main.article") where the HTML contains no <main class="article"> element; using a selector valid for the rendered DOM but the passed html is pre-JavaScript markup; passing a selector targeting elements inside <head> (body.select never sees them); passing malformed selectors like "div[" that BeautifulSoup cannot parse.

Common situations: Scraping SPAs where content is injected by JS after the static HTML was fetched; site redesigns that rename container classes; typos in configured selectors (e.g. missing dot for class); pages that return a captcha/consent wall instead of expected content so the selector matches nothing.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/d11d2519e95c5d70. Report an issue: GitHub.

Appendix: source

Thrown at crawl4ai/utils.py:925

    Returns:
        Dict[str, Any]: Extracted content including Markdown, cleaned HTML, media, links, and metadata.
    """

    try:
        if not html:
            return None
        # Parse HTML content with BeautifulSoup
        soup = BeautifulSoup(html, "html.parser")

        # Get the content within the <body> tag
        body = soup.body

        # If css_selector is provided, extract content based on the selector
        if css_selector:
            selected_elements = body.select(css_selector)
            if not selected_elements:
                raise InvalidCSSSelectorError(
                    f"Invalid CSS selector , No elements found for CSS selector: {css_selector}"
                )
            div_tag = soup.new_tag("div")
            for el in selected_elements:
                div_tag.append(el)
            body = div_tag

        links = {"internal": [], "external": []}

        # Extract all internal and external links
        for a in body.find_all("a", href=True):
            href = a["href"]
            url_base = url.split("/")[2]
            if href.startswith("http") and url_base not in href:
                links["external"].append({"href": href, "text": a.get_text()})
            else:
                links["internal"].append({"href": href, "text": a.get_text()})

View on GitHub (pinned to 7e80152142)