rohitg00/ai-engineering-from-scratch · error · ValueError

mode decision needs rationale, repetition, change policy…

Error message

mode decision needs rationale, repetition, change policy, and rejections

What it means

Completeness check on a mode-trials decision block: experiment.decision must carry a non-empty 'why', 'repeatPolicy', 'changePolicy', and a non-empty list of 'rejected' alternatives. One of these is missing or blank, so the experiment's conclusion is not auditable and validation fails.

Solutions

1. Fill in why/repeatPolicy/changePolicy and list the rejected modes with reasons
2. Document the repetition count and what would change the chosen mode
3. Treat the decision block as mandatory in the schema and validate it before summarizing

When it happens

Trigger: Thrown at certifications/claude/lessons/02-model-selection-and-token-economics/code/main.py:213 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/23fa99d2136027c2. Report an issue: GitHub.

Appendix: source

Thrown at certifications/claude/lessons/02-model-selection-and-token-economics/code/main.py:213

        if selected != best["id"]:
            errors.append("selectedConfiguration must be the least costly passing mode")

    decision = experiment.get("decision")
    if (
        not isinstance(decision, dict)
        or not str(decision.get("why", "")).strip()
        or not str(decision.get("repeatPolicy", "")).strip()
        or not str(decision.get("changePolicy", "")).strip()
        or not _nonempty_strings(decision.get("rejected"))
    ):
        errors.append("mode decision needs rationale, repetition, change policy, and rejections")
    return errors


def summarize_mode_trials(experiment: dict[str, Any]) -> dict[str, Any]:
    errors = validate_mode_trials(experiment)
    if errors:
        raise ValueError("; ".join(errors))
    selected = next(
        item for item in experiment["configurations"] if item["id"] == experiment["selectedConfiguration"]
    )
    return {
        "selectedConfiguration": selected["id"],
        "settings": selected["settings"],
        "minimumQuality": selected["summary"]["minimumQuality"],
        "p95LatencyMs": selected["summary"]["p95LatencyMs"],
        "meanCostUnits": selected["summary"]["meanCostUnits"],
        "supportedConfigurations": sum(
            item["support"]["status"] == "docs-supported" for item in experiment["configurations"]
        ),
    }


def _validate_repeated_runs(configuration_id: str, runs: Any, minimum_run_count: int) -> list[str]:
    if not isinstance(runs, list) or len(runs) < minimum_run_count:
        return [f"{configuration_id} requires at least {minimum_run_count} repeated runs"]

View on GitHub (pinned to 39ea8a1c6d)