unclecode/crawl4ai · error · Exception

Failed to generate schema: {str(e)}

Error message

Failed to generate schema: {str(e)}

What it means

Catch-all raised by the generate_schema loop's final except Exception: it wraps any non-JSONDecodeError failure during a generation attempt — LLM API errors (auth, rate limit, connectivity), empty-response ValueError rethrown here, or unexpected runtime errors — prefixed with 'Failed to generate schema:'.

Source

Thrown at crawl4ai/extraction_strategy.py:1946

                    usage.total_tokens += response.usage.total_tokens
                raw = response.choices[0].message.content
                if not raw or not raw.strip():
                    raise ValueError("LLM returned an empty response")

                schema = json.loads(_strip_markdown_fences(raw))
                last_schema = schema
            except json.JSONDecodeError as e:
                # JSON parse failure — ask LLM to fix it
                if not validate or attempt >= max_attempts - 1:
                    raise Exception(f"Failed to parse schema JSON: {str(e)}")
                messages.append({"role": "assistant", "content": raw})
                messages.append({"role": "user", "content": (
                    f"Your response was not valid JSON. Parse error: {e}\n"
                    "Please return ONLY valid JSON, nothing else."
                )})
                continue
            except Exception as e:
                raise Exception(f"Failed to generate schema: {str(e)}")

            # If validation is off, return immediately (zero overhead path)
            if not validate:
                return schema

            # --- Validation feedback loop ---
            # Validate against original HTML(s); success if works on at least one
            best_result = None
            for orig_html in original_htmls:
                vr = JsonElementExtractionStrategy._validate_schema(
                    schema, orig_html, schema_type,
                    expected_fields=expected_fields,
                )
                if best_result is None or vr["populated_fields"] > best_result["populated_fields"]:
                    best_result = vr
                if vr["success"]:
                    break

View on GitHub (pinned to 7e80152142)

Solutions

  1. Read the wrapped message — '401', 'rate limit', 'Connection' each point to a different fix (key, backoff/delay, network)
  2. Verify llm_config credentials with a minimal completion call before generating schemas
  3. Retry with backoff around generate_schema; transient 429/5xx provider errors usually clear
  4. For local servers, check server logs at the failure timestamp

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema(html=html, llm_config=cfg)
# Failed to generate schema: Error code: 429 ...

// after
import asyncio
for i in range(5):
    try:
        schema = await JsonElementExtractionStrategy.generate_schema(html=html, llm_config=cfg)
        break
    except Exception as e:
        if "429" not in str(e) or i == 4:
            raise
        await asyncio.sleep(2 ** i)
Defensive patterns

Strategy: retry

Validate before calling

from crawl4ai import LLMConfig

# minimal credential/API check before generating
probe_cfg = LLMConfig(provider=cfg.provider, api_token=cfg.api_token, base_url=cfg.base_url)
resp = await aperform_completion_with_backoff(
    provider=probe_cfg.provider, api_token=probe_cfg.api_token,
    prompt='Reply: OK', base_url=probe_cfg.base_url)
# raises early with a clear provider error if credentials/endpoint are bad

Try / catch

import asyncio

for i in range(4):
    try:
        schema = await JsonElementExtractionStrategy.generate_schema(html=h, llm_config=cfg)
        break
    except Exception as e:
        msg = str(e)
        if not msg.startswith("Failed to generate schema:") or i == 3:
            raise
        if "429" in msg or "5" in msg.split(':')[1][:1]:  # rate limit / 5xx -> backoff
            await asyncio.sleep(2 ** i)
        else:
            raise

Prevention

When it happens

Trigger: LLM provider returns 401/429/500 during aperform_completion_with_backoff; the 'LLM returned an empty response' ValueError from earlier in the try block; network drop mid-request. The original exception's message survives in str(e), so it is the diagnostic handle.

Common situations: Invalid or expired API key in llm_config; rate limits hit because generation makes several calls per attempt (generation + validation); flaky network to the provider; local LLM server crashing under the schema prompt.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/aa225d0b05ce35da. Report an issue: GitHub.