What causes "Semantic code generation failed: {str(e)}" in ScrapeGraphAI/Scrapegraph-ai?

Check the embedded {str(e)} text — it carries the root cause (auth, rate limit, parse error)

ScrapeGraphAI/Scrapegraph-ai · error · CodeGenerationError

Semantic code generation failed: {str(e)}

Error message

Semantic code generation failed: {str(e)}

What it means

CodeGenerationError raised by semantic_focused_code_generation when an unexpected exception occurs during LLM-driven code generation/correction. The original exception text is embedded in the message. It signals the code-correction loop could not produce or regenerate scraping code.

Source

Thrown at scrapegraphai/utils/code_error_correction.py:319

        return chain.invoke(
            {
                "analysis": analysis,
                "generated_code": validated_state.generated_code,
                "generated_result": json.dumps(
                    validated_state.execution_result, indent=2
                ),
                "reference_result": json.dumps(
                    validated_state.reference_answer, indent=2
                ),
            }
        )

    except KeyError as e:
        raise InvalidCorrectionStateError(
            f"Missing required key in state dictionary: {e}"
        )
    except Exception as e:
        raise CodeGenerationError(f"Semantic code generation failed: {str(e)}")

View on GitHub (pinned to 532dfffbf6)

Solutions

Check the embedded {str(e)} text — it carries the root cause (auth, rate limit, parse error)
Verify the llm_model config (API key, model name) used for the correction step
Catch CodeGenerationError in the loop and fall back to the original code instead of retrying
Reproduce with verbose logging enabled to see the underlying LLM error

Example fix

// before
new_code = semantic_focused_code_generation(llm, code, corrections)
// after
try:
    new_code = semantic_focused_code_generation(llm, code, corrections)
except CodeGenerationError as e:
    logger.error("correction failed: %s", e)
    new_code = code  # keep original

Defensive patterns

Strategy: try-catch

Validate before calling

from scrapegraphai.utils.code_error_correction import CodeGenerationError
# ensure llm model config is valid before running the loop
assert llm_model and getattr(llm_model, "model", None), "invalid llm config"

Try / catch

try:
    code = semantic_focused_code_generation(llm, code, corrections)
except CodeGenerationError as e:
    logger.warning("keeping original code: %s", e)
except InvalidCorrectionStateError as e:
    logger.error("bad state: %s", e)
    raise

Prevention

Validate LLM credentials/model before enabling the correction loop
Always keep the last working code as a fallback
Log the embedded cause message for diagnosis

When it happens

Trigger: Calling semantic_comparison_loop / semantic_focused_code_generation when the LLM call fails, the model response cannot be parsed, or any non-KeyError exception escapes inside the generation routine.

Common situations: Invalid or missing LLM API key, wrong model name, rate limits, or malformed prompt/schema passed to the correction loop after a scraping failure.

Related errors

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/9c804bb52d5eeb71. Report an issue: GitHub.