{"record":{"id":"0a9c16d6cd116507","repo":"shareAI-lab/learn-claude-code","slug":"goal-evaluator-returned-invalid-json","errorCode":null,"errorMessage":"goal evaluator returned invalid JSON","messagePattern":"goal evaluator returned invalid JSON","errorType":"exception","errorClass":"GoalError","httpStatus":null,"severity":"error","filePath":"s17_goal_loop/code.py","lineNumber":183,"sourceCode":"            break\n        selected.append(item)\n        size += item_size\n    return \"\\n\\n\".join(reversed(selected))\n\n\ndef _parse_json_object(text: str) -> dict[str, Any]:\n    stripped = text.strip()\n    if stripped.startswith(\"```\"):\n        lines = stripped.splitlines()\n        if lines and lines[0].startswith(\"```\"):\n            lines = lines[1:]\n        if lines and lines[-1].strip() == \"```\":\n            lines = lines[:-1]\n        stripped = \"\\n\".join(lines).strip()\n    try:\n        value = json.loads(stripped)\n    except json.JSONDecodeError as error:\n        raise GoalError(\"goal evaluator returned invalid JSON\") from error\n    if not isinstance(value, dict):\n        raise GoalError(\"goal evaluator must return a JSON object\")\n    if not isinstance(value.get(\"ok\"), bool):\n        raise GoalError(\"goal evaluator response requires boolean 'ok'\")\n    if not isinstance(value.get(\"reason\"), str) or not value[\"reason\"].strip():\n        raise GoalError(\"goal evaluator response requires non-empty 'reason'\")\n    impossible = value.get(\"impossible\", False)\n    if not isinstance(impossible, bool):\n        raise GoalError(\"goal evaluator 'impossible' must be boolean\")\n    if value[\"ok\"] and impossible:\n        raise GoalError(\n            \"goal evaluator cannot return both ok and impossible\"\n        )\n    return {\n        \"ok\": value[\"ok\"],\n        \"reason\": value[\"reason\"].strip(),\n        \"impossible\": impossible,\n    }","sourceCodeStart":165,"sourceCodeEnd":201,"githubUrl":"https://github.com/shareAI-lab/learn-claude-code/blob/985456f4adea6f4df8fbad4112245dbd97444eae/s17_goal_loop/code.py#L165-L201","documentation":"The goal loop's evaluator response parser (_parse_json_object) strips markdown code fences and then json.loads the text; a JSONDecodeError is re-raised as GoalError('goal evaluator returned invalid JSON'). The evaluator model must reply with a parseable JSON object, and this is the first gate it must pass.","triggerScenarios":"A PromptGoalEvaluator (or any GoalEvaluator implementation whose text feeds _parse_json_object) returns prose, truncated JSON, or empty text after fence stripping, so json.loads fails.","commonSituations":"Small evaluator token budget (DEFAULT_EVALUATOR_MAX_TOKENS=512) truncating the JSON mid-object; the model wrapping output in prose or double fences; non-JSON evaluator backends wired in.","solutions":["Raise the evaluator's max_tokens so its JSON response is never truncated.","Strengthen the evaluator system prompt: 'Respond with ONLY a JSON object, no prose, no code fences.'","If using a custom evaluator class, make it return the raw JSON text exactly (no logging prefixes, no commentary).","Retry the evaluation — transient model malformation is common; the parse is deterministic once output is well-formed."],"exampleFix":"# before\nevaluator = PromptGoalEvaluator(client, max_tokens=100)\n\n# after\nevaluator = PromptGoalEvaluator(client, max_tokens=2048)\n# system prompt: 'Output only {\"ok\": bool, \"reason\": str, \"impossible\": bool}'","handlingStrategy":"retry","validationCode":"import json\ndef evaluator_text_is_json(text: str) -> bool:\n    try:\n        json.loads(text.strip().strip('`'))\n        return True\n    except json.JSONDecodeError:\n        return False","typeGuard":null,"tryCatchPattern":"for attempt in range(2):\n    text = await evaluator.evaluate(messages)\n    try:\n        return _parse_json_object(text)\n    except GoalError as e:\n        if \"invalid JSON\" not in str(e):\n            raise\n        messages = messages + [{\"role\": \"user\", \"content\": \"Return ONLY the JSON object.\"}]\nraise GoalError(\"evaluator failed to produce JSON after retry\")","preventionTips":["Give the evaluator a strict system prompt: JSON object only, no fences, no prose.","Set evaluator max_tokens comfortably above the expected response size (default 512 is tight).","Use structured-output/JSON mode on the evaluator model if the API offers it."],"tags":["goal-loop","evaluator","json","llm","parsing"],"backgroundTag":null,"analyzedSha":"985456f4adea6f4df8fbad4112245dbd97444eae","analyzedAt":"2026-08-14T22:02:26.028Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}