unclecode/crawl4ai · error · Exception

Failed to parse schema JSON: {str(e)}

Error message

Failed to parse schema JSON: {str(e)}

What it means

Raised when the LLM's schema response cannot be parsed as JSON even after stripping markdown fences, AND either validate=False (self-repair loop disabled) or the final attempt (attempt >= max_attempts - 1) was reached. The json.JSONDecodeError detail is embedded so you can see where parsing broke.

Source

Thrown at crawl4ai/extraction_strategy.py:1938

                    api_token=llm_config.api_token,
                    base_url=llm_config.base_url,
                    messages=messages,
                    extra_args=kwargs,
                )
                if usage is not None:
                    usage.completion_tokens += response.usage.completion_tokens
                    usage.prompt_tokens += response.usage.prompt_tokens
                    usage.total_tokens += response.usage.total_tokens
                raw = response.choices[0].message.content
                if not raw or not raw.strip():
                    raise ValueError("LLM returned an empty response")

                schema = json.loads(_strip_markdown_fences(raw))
                last_schema = schema
            except json.JSONDecodeError as e:
                # JSON parse failure — ask LLM to fix it
                if not validate or attempt >= max_attempts - 1:
                    raise Exception(f"Failed to parse schema JSON: {str(e)}")
                messages.append({"role": "assistant", "content": raw})
                messages.append({"role": "user", "content": (
                    f"Your response was not valid JSON. Parse error: {e}\n"
                    "Please return ONLY valid JSON, nothing else."
                )})
                continue
            except Exception as e:
                raise Exception(f"Failed to generate schema: {str(e)}")

            # If validation is off, return immediately (zero overhead path)
            if not validate:
                return schema

            # --- Validation feedback loop ---
            # Validate against original HTML(s); success if works on at least one
            best_result = None
            for orig_html in original_htmls:
                vr = JsonElementExtractionStrategy._validate_schema(

View on GitHub (pinned to 7e80152142)

Solutions

  1. Keep validate=True (default) so the built-in repair loop can re-ask the LLM
  2. Increase max_attempts and the model's token budget so truncation stops producing half-parsed JSON
  3. Switch to a more instruction-following model (e.g. a flagship chat model) for schema generation
  4. If validate=False, catch the exception and fall back to parsing the schema manually from the raw response if you captured it

Example fix

// before
schema = await JsonElementExtractionStrategy.generate_schema(
    html=html, llm_config=cfg, validate=False)  # chatty model -> parse error

// after
schema = await JsonElementExtractionStrategy.generate_schema(
    html=html, llm_config=cfg, validate=True, max_attempts=4)
Defensive patterns

Strategy: retry

Try / catch

try:
    schema = await JsonElementExtractionStrategy.generate_schema(
        html=h, llm_config=cfg, validate=True, max_attempts=4)
except Exception as e:
    if "Failed to parse schema JSON" in str(e):
        # regenerate with a stricter, larger model or simplified HTML sample
        schema = await JsonElementExtractionStrategy.generate_schema(
            html=simplified_html, llm_config=better_cfg,
            validate=True, max_attempts=4)
    else:
        raise

Prevention

When it happens

Trigger: LLM returns prose around the JSON that _strip_markdown_fences cannot remove (unfenced code, leading commentary); truncated JSON from token limits; validate=False with a chatty model that always wraps JSON in text; max_attempts exhausted after repeated 'return ONLY valid JSON' corrections.

Common situations: Small models that narrate before/after JSON; max_tokens too low so the schema is cut off mid-object; validate=False chosen for speed with a model that is not strict-JSON capable.

Understand the failure class

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/f193f5422ccba5e7. Report an issue: GitHub.