unclecode/crawl4ai · error · InvalidCSSSelectorError

Invalid CSS selector , No elements found for CSS selector: {

Error message

Invalid CSS selector , No elements found for CSS selector: {css_selector}

What it means

Raised by get_content_of_website (crawl4ai/utils.py:925) as InvalidCSSSelectorError when a css_selector was supplied to the HTML extraction helper but BeautifulSoup's body.select(css_selector) returned zero elements. Note the message is slightly misleading: the selector may be syntactically valid; it simply matched nothing in the parsed <body> (or the page has no <body>, making body None and the select fail). It exists so callers learn their extraction selector did not match the fetched HTML.

Source

Thrown at crawl4ai/utils.py:925

    Returns:
        Dict[str, Any]: Extracted content including Markdown, cleaned HTML, media, links, and metadata.
    """

    try:
        if not html:
            return None
        # Parse HTML content with BeautifulSoup
        soup = BeautifulSoup(html, "html.parser")

        # Get the content within the <body> tag
        body = soup.body

        # If css_selector is provided, extract content based on the selector
        if css_selector:
            selected_elements = body.select(css_selector)
            if not selected_elements:
                raise InvalidCSSSelectorError(
                    f"Invalid CSS selector , No elements found for CSS selector: {css_selector}"
                )
            div_tag = soup.new_tag("div")
            for el in selected_elements:
                div_tag.append(el)
            body = div_tag

        links = {"internal": [], "external": []}

        # Extract all internal and external links
        for a in body.find_all("a", href=True):
            href = a["href"]
            url_base = url.split("/")[2]
            if href.startswith("http") and url_base not in href:
                links["external"].append({"href": href, "text": a.get_text()})
            else:
                links["internal"].append({"href": href, "text": a.get_text()})

View on GitHub (pinned to 7e80152142)

Solutions

Verify the selector against the exact HTML string being passed (open it or print soup.body.prettify()) — the DOM you select on is the static parsed body, not the rendered page.
Loosen the selector ("article" instead of "div.article-content.prose") or provide a fallback list of selectors tried in order.
For JS-rendered content, fetch via AsyncPlaywrightCrawlerStrategy / the main AsyncWebcrawler with js_code or wait_for, then pass the rendered HTML (or use CrawlerRunConfig.css_selector which handles this in the pipeline).
Validate the selector syntax first with soup.select on the soup object and catch InvalidCSSSelectorError to degrade gracefully.

Example fix

// before
result = get_content_of_website(url, html, css_selector="main.post-content")

// after
from crawl4ai.utils import InvalidCSSSelectorError
for sel in ("main.post-content", "article", "body"):
    try:
        result = get_content_of_website(url, html, css_selector=sel)
        break
    except InvalidCSSSelectorError:
        continue

Defensive patterns

Strategy: validation

Validate before calling

from bs4 import BeautifulSoup

def selector_matches(html: str, css_selector: str) -> bool:
    soup = BeautifulSoup(html, "html.parser")
    body = soup.body or soup
    try:
        return bool(body.select(css_selector))
    except Exception:
        return False

Try / catch

from crawl4ai.utils import InvalidCSSSelectorError
try:
    content = get_content_of_website(url, html, css_selector=sel)
except InvalidCSSSelectorError as e:
    logger.warning("selector %r matched nothing: %s", sel, e)
    content = get_content_of_website(url, html)  # fallback: full body

Prevention

Validate selectors against the fetched static HTML before extraction
Keep a prioritized list of fallback selectors per site template
Log the saved HTML when a selector fails to catch site-redesign drift early

When it happens

Trigger: Calling get_content_of_website(url, html, css_selector="main.article") where the HTML contains no <main class="article"> element; using a selector valid for the rendered DOM but the passed html is pre-JavaScript markup; passing a selector targeting elements inside <head> (body.select never sees them); passing malformed selectors like "div[" that BeautifulSoup cannot parse.

Common situations: Scraping SPAs where content is injected by JS after the static HTML was fetched; site redesigns that rename container classes; typos in configured selectors (e.g. missing dot for class); pages that return a captcha/consent wall instead of expected content so the selector matches nothing.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/d11d2519e95c5d70. Report an issue: GitHub.