{"record":{"id":"aa225d0b05ce35da","repo":"unclecode/crawl4ai","slug":"failed-to-generate-schema-str-e","errorCode":null,"errorMessage":"Failed to generate schema: {str(e)}","messagePattern":"Failed to generate schema: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crawl4ai/extraction_strategy.py","lineNumber":1946,"sourceCode":"                    usage.total_tokens += response.usage.total_tokens\n                raw = response.choices[0].message.content\n                if not raw or not raw.strip():\n                    raise ValueError(\"LLM returned an empty response\")\n\n                schema = json.loads(_strip_markdown_fences(raw))\n                last_schema = schema\n            except json.JSONDecodeError as e:\n                # JSON parse failure — ask LLM to fix it\n                if not validate or attempt >= max_attempts - 1:\n                    raise Exception(f\"Failed to parse schema JSON: {str(e)}\")\n                messages.append({\"role\": \"assistant\", \"content\": raw})\n                messages.append({\"role\": \"user\", \"content\": (\n                    f\"Your response was not valid JSON. Parse error: {e}\\n\"\n                    \"Please return ONLY valid JSON, nothing else.\"\n                )})\n                continue\n            except Exception as e:\n                raise Exception(f\"Failed to generate schema: {str(e)}\")\n\n            # If validation is off, return immediately (zero overhead path)\n            if not validate:\n                return schema\n\n            # --- Validation feedback loop ---\n            # Validate against original HTML(s); success if works on at least one\n            best_result = None\n            for orig_html in original_htmls:\n                vr = JsonElementExtractionStrategy._validate_schema(\n                    schema, orig_html, schema_type,\n                    expected_fields=expected_fields,\n                )\n                if best_result is None or vr[\"populated_fields\"] > best_result[\"populated_fields\"]:\n                    best_result = vr\n                if vr[\"success\"]:\n                    break\n","sourceCodeStart":1928,"sourceCodeEnd":1964,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/extraction_strategy.py#L1928-L1964","documentation":"Catch-all raised by the generate_schema loop's final except Exception: it wraps any non-JSONDecodeError failure during a generation attempt — LLM API errors (auth, rate limit, connectivity), empty-response ValueError rethrown here, or unexpected runtime errors — prefixed with 'Failed to generate schema:'.","triggerScenarios":"LLM provider returns 401/429/500 during aperform_completion_with_backoff; the 'LLM returned an empty response' ValueError from earlier in the try block; network drop mid-request. The original exception's message survives in str(e), so it is the diagnostic handle.","commonSituations":"Invalid or expired API key in llm_config; rate limits hit because generation makes several calls per attempt (generation + validation); flaky network to the provider; local LLM server crashing under the schema prompt.","solutions":["Read the wrapped message — '401', 'rate limit', 'Connection' each point to a different fix (key, backoff/delay, network)","Verify llm_config credentials with a minimal completion call before generating schemas","Retry with backoff around generate_schema; transient 429/5xx provider errors usually clear","For local servers, check server logs at the failure timestamp"],"exampleFix":"// before\nschema = await JsonElementExtractionStrategy.generate_schema(html=html, llm_config=cfg)\n# Failed to generate schema: Error code: 429 ...\n\n// after\nimport asyncio\nfor i in range(5):\n    try:\n        schema = await JsonElementExtractionStrategy.generate_schema(html=html, llm_config=cfg)\n        break\n    except Exception as e:\n        if \"429\" not in str(e) or i == 4:\n            raise\n        await asyncio.sleep(2 ** i)","handlingStrategy":"retry","validationCode":"from crawl4ai import LLMConfig\n\n# minimal credential/API check before generating\nprobe_cfg = LLMConfig(provider=cfg.provider, api_token=cfg.api_token, base_url=cfg.base_url)\nresp = await aperform_completion_with_backoff(\n    provider=probe_cfg.provider, api_token=probe_cfg.api_token,\n    prompt='Reply: OK', base_url=probe_cfg.base_url)\n# raises early with a clear provider error if credentials/endpoint are bad","typeGuard":null,"tryCatchPattern":"import asyncio\n\nfor i in range(4):\n    try:\n        schema = await JsonElementExtractionStrategy.generate_schema(html=h, llm_config=cfg)\n        break\n    except Exception as e:\n        msg = str(e)\n        if not msg.startswith(\"Failed to generate schema:\") or i == 3:\n            raise\n        if \"429\" in msg or \"5\" in msg.split(':')[1][:1]:  # rate limit / 5xx -> backoff\n            await asyncio.sleep(2 ** i)\n        else:\n            raise","preventionTips":["Validate LLM credentials with a tiny completion before batch schema work","Read the wrapped message — it names the real API failure","Back off on 429/5xx; generation makes several LLM calls per attempt"],"tags":["extraction","schema-generation","llm","api-error"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}