unclecode/crawl4ai · error · ValueError

Either 'html' or 'url' must be provided

Error message

Either 'html' or 'url' must be provided

What it means

Raised by JsonElementExtractionStrategy.generate_schema when both html is None and url is None (or url is an empty list). The schema generator needs at least one page of real HTML to infer selectors from, so it refuses to run without sample content.

Source

Thrown at crawl4ai/extraction_strategy.py:1813

            validate (bool): If True, validate the schema against the HTML and
                refine via LLM feedback loop. Defaults to False (zero overhead).
            max_refinements (int): Max refinement rounds when validate=True. Defaults to 3.
            usage (TokenUsage, optional): Token usage accumulator. If provided,
                token counts from all LLM calls (including inference and
                validation retries) are added to it in-place.
            **kwargs: Additional args passed to LLM processor.

        Returns:
            dict: Generated schema following the JsonElementExtractionStrategy format.

        Raises:
            ValueError: If neither html nor url is provided.
        """
        from .utils import aperform_completion_with_backoff, preprocess_html_for_schema

        # Validate inputs
        if html is None and (url is None or (isinstance(url, list) and len(url) == 0)):
            raise ValueError("Either 'html' or 'url' must be provided")

        # Check deprecated parameters
        for name, message in JsonElementExtractionStrategy._GENERATE_SCHEMA_UNWANTED_PROPS.items():
            if locals()[name] is not None:
                raise AttributeError(f"Setting '{name}' is deprecated. {message}")

        if llm_config is None:
            llm_config = create_llm_config()

        # Save original HTML(s) before preprocessing (for validation against real HTML)
        original_htmls = []

        # Fetch HTML from URL(s) if provided
        if url is not None:
            from .async_webcrawler import AsyncWebCrawler
            from .async_configs import BrowserConfig, CrawlerRunConfig, CacheMode

            browser_config = BrowserConfig(

View on GitHub (pinned to 7e80152142)

Solutions

  1. Pass either html="<html>...</html>" or a non-empty url / list of URLs
  2. If loading content dynamically, assert the value is truthy before calling generate_schema
  3. When accepting user input, validate that at least one of the two parameters is populated

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema()  # ValueError

// after
schema = await JsonElementExtractionStrategy.generate_schema(url="https://example.com/products")
Defensive patterns

Strategy: validation

Validate before calling

has_html = bool(html and html.strip())
has_url = bool(url) and not (isinstance(url, list) and len(url) == 0)
if not (has_html or has_url):
    raise ValueError("provide html= or a non-empty url= before generating a schema")

Type guard

def has_schema_input(html, url) -> bool:
    if html and html.strip():
        return True
    if isinstance(url, str) and url:
        return True
    return isinstance(url, list) and len(url) > 0

Prevention

When it happens

Trigger: Calling generate_schema() with no arguments; passing url=[] (an empty list passes the isinstance check and len==0 branch); passing url=None and forgetting html; conditionally building arguments where both branches end up None.

Common situations: Dynamic pipelines where the URL list was filtered to empty before the call; refactoring code that used to pass html but now passes a variable that is None on some paths; notebook prototyping with placeholder arguments.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/ee8ce018792faeb1. Report an issue: GitHub.