{"record":{"id":"62007077d3399730","repo":"shareAI-lab/learn-claude-code","slug":"goal-evaluator-cannot-return-both-ok-and-impossibl","errorCode":null,"errorMessage":"goal evaluator cannot return both ok and impossible","messagePattern":"goal evaluator cannot return both ok and impossible","errorType":"validation","errorClass":"GoalError","httpStatus":null,"severity":"error","filePath":"s17_goal_loop/code.py","lineNumber":194,"sourceCode":"            lines = lines[1:]\n        if lines and lines[-1].strip() == \"```\":\n            lines = lines[:-1]\n        stripped = \"\\n\".join(lines).strip()\n    try:\n        value = json.loads(stripped)\n    except json.JSONDecodeError as error:\n        raise GoalError(\"goal evaluator returned invalid JSON\") from error\n    if not isinstance(value, dict):\n        raise GoalError(\"goal evaluator must return a JSON object\")\n    if not isinstance(value.get(\"ok\"), bool):\n        raise GoalError(\"goal evaluator response requires boolean 'ok'\")\n    if not isinstance(value.get(\"reason\"), str) or not value[\"reason\"].strip():\n        raise GoalError(\"goal evaluator response requires non-empty 'reason'\")\n    impossible = value.get(\"impossible\", False)\n    if not isinstance(impossible, bool):\n        raise GoalError(\"goal evaluator 'impossible' must be boolean\")\n    if value[\"ok\"] and impossible:\n        raise GoalError(\n            \"goal evaluator cannot return both ok and impossible\"\n        )\n    return {\n        \"ok\": value[\"ok\"],\n        \"reason\": value[\"reason\"].strip(),\n        \"impossible\": impossible,\n    }\n\n\nclass PromptGoalEvaluator:\n    \"\"\"A separate, tool-free model that judges the transcript.\"\"\"\n\n    def __init__(\n        self,\n        client: Any,\n        model: str,\n        max_tokens: int = DEFAULT_EVALUATOR_MAX_TOKENS,\n    ):","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/shareAI-lab/learn-claude-code/blob/985456f4adea6f4df8fbad4112245dbd97444eae/s17_goal_loop/code.py#L176-L212","documentation":"The goal evaluator (a separate prompt-based model that judges the transcript) returned a JSON object whose 'ok' field is true while its 'impossible' field is also true. These are contradictory verdicts: 'ok' means the goal is met, 'impossible' means the goal can never be met. The library validates the evaluator's JSON response in _validate (s17_goal_loop/code.py:194) and rejects this combination before it can corrupt goal state.","triggerScenarios":"A PromptGoalEvaluator model call returns a response such as {\"ok\": true, \"reason\": \"done\", \"impossible\": true} — usually from a weak or confused evaluator model, an ambiguous prompt, or output that happens to set both flags. Any session.run()/query cycle that invokes the evaluator can surface it.","commonSituations":"Using a small/cheap model (e.g. a haiku-tier default from GOAL_EVALUATOR_MODEL_ID or ANTHROPIC_DEFAULT_HAIKU_MODEL) as the evaluator; ambiguous goal conditions the model hedges on; prompt templates for the evaluator that do not forbid setting both flags; JSON produced by the model being partially malformed semantically even though it parses.","solutions":["Set GOAL_EVALUATOR_MODEL_ID to a stronger model so verdicts are consistent","Tighten the evaluator system prompt: explicitly state 'never set both ok and impossible; impossible only when ok is false'","Rephrase the goal condition to be concrete and verifiable so the evaluator is not forced to hedge","Wrap the session query in try/except GoalError and retry or fall back to manual judgment"],"exampleFix":"# before\ngoal.set_goal(\"make the app good\")\n\n# after\ngoal.set_goal(\"all tests in tests/ pass when run with pytest\")","handlingStrategy":"retry","validationCode":"verdict = json.loads(evaluator_output)\nif not isinstance(verdict, dict):\n    raise ValueError(\"evaluator output is not an object\")\nif verdict.get(\"ok\") is True and verdict.get(\"impossible\", False) is True:\n    # contradictory verdict: re-ask the evaluator instead of passing it through\n    verdict = reask_evaluator(strict_prompt=True)","typeGuard":"def is_consistent_verdict(v: object) -> bool:\n    return (\n        isinstance(v, dict)\n        and isinstance(v.get(\"ok\"), bool)\n        and isinstance(v.get(\"impossible\", False), bool)\n        and not (v[\"ok\"] and v[\"impossible\"])\n    )","tryCatchPattern":"try:\n    result = await session.submit(query)\nexcept GoalError as error:\n    if \"cannot return both ok and impossible\" in str(error):\n        result = await session.submit(query)  # one retry; verdicts usually converge\n    else:\n        raise","preventionTips":["Pin GOAL_EVALUATOR_MODEL_ID to a capable model rather than the cheapest default","State in the evaluator prompt that ok and impossible are mutually exclusive","Keep goal conditions concrete so the evaluator never needs to hedge"],"tags":["goal-loop","evaluator","json-validation","llm"],"backgroundTag":null,"analyzedSha":"985456f4adea6f4df8fbad4112245dbd97444eae","analyzedAt":"2026-08-14T22:02:26.028Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}