{"record":{"id":"f193f5422ccba5e7","repo":"unclecode/crawl4ai","slug":"failed-to-parse-schema-json-str-e","errorCode":null,"errorMessage":"Failed to parse schema JSON: {str(e)}","messagePattern":"Failed to parse schema JSON: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crawl4ai/extraction_strategy.py","lineNumber":1938,"sourceCode":"                    api_token=llm_config.api_token,\n                    base_url=llm_config.base_url,\n                    messages=messages,\n                    extra_args=kwargs,\n                )\n                if usage is not None:\n                    usage.completion_tokens += response.usage.completion_tokens\n                    usage.prompt_tokens += response.usage.prompt_tokens\n                    usage.total_tokens += response.usage.total_tokens\n                raw = response.choices[0].message.content\n                if not raw or not raw.strip():\n                    raise ValueError(\"LLM returned an empty response\")\n\n                schema = json.loads(_strip_markdown_fences(raw))\n                last_schema = schema\n            except json.JSONDecodeError as e:\n                # JSON parse failure — ask LLM to fix it\n                if not validate or attempt >= max_attempts - 1:\n                    raise Exception(f\"Failed to parse schema JSON: {str(e)}\")\n                messages.append({\"role\": \"assistant\", \"content\": raw})\n                messages.append({\"role\": \"user\", \"content\": (\n                    f\"Your response was not valid JSON. Parse error: {e}\\n\"\n                    \"Please return ONLY valid JSON, nothing else.\"\n                )})\n                continue\n            except Exception as e:\n                raise Exception(f\"Failed to generate schema: {str(e)}\")\n\n            # If validation is off, return immediately (zero overhead path)\n            if not validate:\n                return schema\n\n            # --- Validation feedback loop ---\n            # Validate against original HTML(s); success if works on at least one\n            best_result = None\n            for orig_html in original_htmls:\n                vr = JsonElementExtractionStrategy._validate_schema(","sourceCodeStart":1920,"sourceCodeEnd":1956,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/extraction_strategy.py#L1920-L1956","documentation":"Raised when the LLM's schema response cannot be parsed as JSON even after stripping markdown fences, AND either validate=False (self-repair loop disabled) or the final attempt (attempt >= max_attempts - 1) was reached. The json.JSONDecodeError detail is embedded so you can see where parsing broke.","triggerScenarios":"LLM returns prose around the JSON that _strip_markdown_fences cannot remove (unfenced code, leading commentary); truncated JSON from token limits; validate=False with a chatty model that always wraps JSON in text; max_attempts exhausted after repeated 'return ONLY valid JSON' corrections.","commonSituations":"Small models that narrate before/after JSON; max_tokens too low so the schema is cut off mid-object; validate=False chosen for speed with a model that is not strict-JSON capable.","solutions":["Keep validate=True (default) so the built-in repair loop can re-ask the LLM","Increase max_attempts and the model's token budget so truncation stops producing half-parsed JSON","Switch to a more instruction-following model (e.g. a flagship chat model) for schema generation","If validate=False, catch the exception and fall back to parsing the schema manually from the raw response if you captured it"],"exampleFix":"// before\nschema = await JsonElementExtractionStrategy.generate_schema(\n    html=html, llm_config=cfg, validate=False)  # chatty model -> parse error\n\n// after\nschema = await JsonElementExtractionStrategy.generate_schema(\n    html=html, llm_config=cfg, validate=True, max_attempts=4)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    schema = await JsonElementExtractionStrategy.generate_schema(\n        html=h, llm_config=cfg, validate=True, max_attempts=4)\nexcept Exception as e:\n    if \"Failed to parse schema JSON\" in str(e):\n        # regenerate with a stricter, larger model or simplified HTML sample\n        schema = await JsonElementExtractionStrategy.generate_schema(\n            html=simplified_html, llm_config=better_cfg,\n            validate=True, max_attempts=4)\n    else:\n        raise","preventionTips":["Keep validate=True so the built-in repair loop re-prompts the LLM","Use an instruction-following model for schema generation","Raise token limits so JSON is not truncated mid-object"],"tags":["extraction","schema-generation","llm","json-parse"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}