unclecode/crawl4ai · error · Exception

Failed to generate schema: {str(e)}

Error message

Failed to generate schema: {str(e)}

What it means

This is a catch-all Exception raised by JsonElementExtractionStrategy.generate_schema() when anything goes wrong during the LLM call used to auto-generate an extraction schema. The original exception is stringified into the message, so the real cause (auth failure, rate limit, invalid JSON in the model response, network error) is nested in the text rather than chained via __cause__.

Source

Thrown at crawl4ai/extraction_strategy.py:1946:4725

        Analyze the HTML and generate a JSON schema that follows the specified format. Only output valid JSON schema, nothing else.
        """

        try:
            # Call LLM with backoff handling
            response = perform_completion_with_backoff(
                provider=llm_config.provider,
                prompt_with_variables="\n\n".join([system_message["content"], user_message["content"]]),
                json_response = True,                
                api_token=llm_config.api_token,
                base_url=llm_config.base_url,
                extra_args=kwargs
            )
            
            # Extract and return schema
            return json.loads(response.choices[0].message.content)
            
        except Exception as e:
            raise Exception(f"Failed to generate schema: {str(e)}")

class JsonCssExtractionStrategy(JsonElementExtractionStrategy):
    """
    Concrete implementation of `JsonElementExtractionStrategy` using CSS selectors.

    How it works:
    1. Parses HTML content with BeautifulSoup.
    2. Selects elements using CSS selectors defined in the schema.
    3. Extracts field data and applies transformations as defined.

    Attributes:
        schema (Dict[str, Any]): The schema defining the extraction rules.
        verbose (bool): Enables verbose logging for debugging purposes.

    Methods:
        _parse_html(html_content): Parses HTML content into a BeautifulSoup object.
        _get_base_elements(parsed_html, selector): Selects base elements using a CSS selector.
        _get_elements(element, selector): Selects child elements using a CSS selector.

View on GitHub (pinned to 7e80152142)

Solutions

  1. Read the text after 'Failed to generate schema:' — it contains the underlying exception; fix that root cause first
  2. Verify llm_config.api_token and llm_config.base_url are correct for the chosen provider (env var set, no typos)
  3. If the model returns non-JSON, retry with a stronger model or re-run — and wrap the call in try/except to log the raw response
  4. For transient network/rate-limit errors, retry with backoff at the call site since the wrapper discards the original exception type

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema(html, llm_config=cfg)
// after
try:
    schema = await JsonElementExtractionStrategy.generate_schema(html, llm_config=cfg)
except Exception as e:
    logger.error("schema generation failed: %s", e)
    schema = None  # fall back to a hand-written CSS/XPath schema
Defensive patterns

Strategy: retry

Validate before calling

import json
from crawl4ai import LLMConfig

def llm_config_ready(cfg: LLMConfig) -> bool:
    return bool(cfg.provider) and bool(cfg.api_token)

Try / catch

import logging
logger = logging.getLogger(__name__)

for attempt in range(3):
    try:
        schema = await JsonElementExtractionStrategy.generate_schema(html, llm_config=cfg)
        break
    except Exception as e:  # wrapper discards original type; match on message
        msg = str(e)
        if attempt < 2 and ("rate" in msg.lower() or "timeout" in msg.lower() or "connection" in msg.lower()):
            await asyncio.sleep(2 ** attempt)
            continue
        logger.error("schema generation failed: %s", msg)
        raise

Prevention

When it happens

Trigger: Calling generate_schema() with an invalid or missing API token in LLMConfig; the LLM returning non-JSON content that fails json.loads(response.choices[0].message.content); provider/base_url misconfiguration; transient network failures during perform_completion_with_backoff.

Common situations: Bad or expired API key in llm_config; pointing base_url at a proxy that does not support the model name; a model that ignores the JSON instruction and returns prose; quota exhaustion or rate limiting on the provider account.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/4bb67ab0550ad9a5. Report an issue: GitHub.