shareAI-lab/learn-claude-code · error · GoalError

goal evaluator cannot return both ok and impossible

Error message

goal evaluator cannot return both ok and impossible

What it means

The goal evaluator (a separate prompt-based model that judges the transcript) returned a JSON object whose 'ok' field is true while its 'impossible' field is also true. These are contradictory verdicts: 'ok' means the goal is met, 'impossible' means the goal can never be met. The library validates the evaluator's JSON response in _validate (s17_goal_loop/code.py:194) and rejects this combination before it can corrupt goal state.

Source

Thrown at s17_goal_loop/code.py:194

            lines = lines[1:]
        if lines and lines[-1].strip() == "```":
            lines = lines[:-1]
        stripped = "\n".join(lines).strip()
    try:
        value = json.loads(stripped)
    except json.JSONDecodeError as error:
        raise GoalError("goal evaluator returned invalid JSON") from error
    if not isinstance(value, dict):
        raise GoalError("goal evaluator must return a JSON object")
    if not isinstance(value.get("ok"), bool):
        raise GoalError("goal evaluator response requires boolean 'ok'")
    if not isinstance(value.get("reason"), str) or not value["reason"].strip():
        raise GoalError("goal evaluator response requires non-empty 'reason'")
    impossible = value.get("impossible", False)
    if not isinstance(impossible, bool):
        raise GoalError("goal evaluator 'impossible' must be boolean")
    if value["ok"] and impossible:
        raise GoalError(
            "goal evaluator cannot return both ok and impossible"
        )
    return {
        "ok": value["ok"],
        "reason": value["reason"].strip(),
        "impossible": impossible,
    }


class PromptGoalEvaluator:
    """A separate, tool-free model that judges the transcript."""

    def __init__(
        self,
        client: Any,
        model: str,
        max_tokens: int = DEFAULT_EVALUATOR_MAX_TOKENS,
    ):

View on GitHub (pinned to 985456f4ad)

Solutions

  1. Set GOAL_EVALUATOR_MODEL_ID to a stronger model so verdicts are consistent
  2. Tighten the evaluator system prompt: explicitly state 'never set both ok and impossible; impossible only when ok is false'
  3. Rephrase the goal condition to be concrete and verifiable so the evaluator is not forced to hedge
  4. Wrap the session query in try/except GoalError and retry or fall back to manual judgment

Example fix

# before
goal.set_goal("make the app good")

# after
goal.set_goal("all tests in tests/ pass when run with pytest")
Defensive patterns

Strategy: retry

Validate before calling

verdict = json.loads(evaluator_output)
if not isinstance(verdict, dict):
    raise ValueError("evaluator output is not an object")
if verdict.get("ok") is True and verdict.get("impossible", False) is True:
    # contradictory verdict: re-ask the evaluator instead of passing it through
    verdict = reask_evaluator(strict_prompt=True)

Type guard

def is_consistent_verdict(v: object) -> bool:
    return (
        isinstance(v, dict)
        and isinstance(v.get("ok"), bool)
        and isinstance(v.get("impossible", False), bool)
        and not (v["ok"] and v["impossible"])
    )

Try / catch

try:
    result = await session.submit(query)
except GoalError as error:
    if "cannot return both ok and impossible" in str(error):
        result = await session.submit(query)  # one retry; verdicts usually converge
    else:
        raise

Prevention

When it happens

Trigger: A PromptGoalEvaluator model call returns a response such as {"ok": true, "reason": "done", "impossible": true} — usually from a weak or confused evaluator model, an ambiguous prompt, or output that happens to set both flags. Any session.run()/query cycle that invokes the evaluator can surface it.

Common situations: Using a small/cheap model (e.g. a haiku-tier default from GOAL_EVALUATOR_MODEL_ID or ANTHROPIC_DEFAULT_HAIKU_MODEL) as the evaluator; ambiguous goal conditions the model hedges on; prompt templates for the evaluator that do not forbid setting both flags; JSON produced by the model being partially malformed semantically even though it parses.

Related errors

AI-assisted analysis of shareAI-lab/learn-claude-code@985456f4ad (2026-08-14). Data as JSON: /api/errors/62007077d3399730. Report an issue: GitHub.