shareAI-lab/learn-claude-code · error · GoalError

goal evaluator returned invalid JSON

Error message

goal evaluator returned invalid JSON

What it means

The goal loop's evaluator response parser (_parse_json_object) strips markdown code fences and then json.loads the text; a JSONDecodeError is re-raised as GoalError('goal evaluator returned invalid JSON'). The evaluator model must reply with a parseable JSON object, and this is the first gate it must pass.

Source

Thrown at s17_goal_loop/code.py:183

            break
        selected.append(item)
        size += item_size
    return "\n\n".join(reversed(selected))


def _parse_json_object(text: str) -> dict[str, Any]:
    stripped = text.strip()
    if stripped.startswith("```"):
        lines = stripped.splitlines()
        if lines and lines[0].startswith("```"):
            lines = lines[1:]
        if lines and lines[-1].strip() == "```":
            lines = lines[:-1]
        stripped = "\n".join(lines).strip()
    try:
        value = json.loads(stripped)
    except json.JSONDecodeError as error:
        raise GoalError("goal evaluator returned invalid JSON") from error
    if not isinstance(value, dict):
        raise GoalError("goal evaluator must return a JSON object")
    if not isinstance(value.get("ok"), bool):
        raise GoalError("goal evaluator response requires boolean 'ok'")
    if not isinstance(value.get("reason"), str) or not value["reason"].strip():
        raise GoalError("goal evaluator response requires non-empty 'reason'")
    impossible = value.get("impossible", False)
    if not isinstance(impossible, bool):
        raise GoalError("goal evaluator 'impossible' must be boolean")
    if value["ok"] and impossible:
        raise GoalError(
            "goal evaluator cannot return both ok and impossible"
        )
    return {
        "ok": value["ok"],
        "reason": value["reason"].strip(),
        "impossible": impossible,
    }

View on GitHub (pinned to 985456f4ad)

Solutions

1. Raise the evaluator's max_tokens so its JSON response is never truncated.
2. Strengthen the evaluator system prompt: 'Respond with ONLY a JSON object, no prose, no code fences.'
3. If using a custom evaluator class, make it return the raw JSON text exactly (no logging prefixes, no commentary).
4. Retry the evaluation — transient model malformation is common; the parse is deterministic once output is well-formed.

Example fix

# before
evaluator = PromptGoalEvaluator(client, max_tokens=100)

# after
evaluator = PromptGoalEvaluator(client, max_tokens=2048)
# system prompt: 'Output only {"ok": bool, "reason": str, "impossible": bool}'

import json
def evaluator_text_is_json(text: str) -> bool:
    try:
        json.loads(text.strip().strip('`'))
        return True
    except json.JSONDecodeError:
        return False

for attempt in range(2):
    text = await evaluator.evaluate(messages)
    try:
        return _parse_json_object(text)
    except GoalError as e:
        if "invalid JSON" not in str(e):
            raise
        messages = messages + [{"role": "user", "content": "Return ONLY the JSON object."}]
raise GoalError("evaluator failed to produce JSON after retry")

When it happens

Trigger: A PromptGoalEvaluator (or any GoalEvaluator implementation whose text feeds _parse_json_object) returns prose, truncated JSON, or empty text after fence stripping, so json.loads fails.

Common situations: Small evaluator token budget (DEFAULT_EVALUATOR_MAX_TOKENS=512) truncating the JSON mid-object; the model wrapping output in prose or double fences; non-JSON evaluator backends wired in.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• goal evaluator must return a JSON object
• goal evaluator response requires boolean 'ok'
• goal evaluator response requires non-empty 'reason'
• goal evaluator 'impossible' must be boolean
• goal evaluator cannot return both ok and impossible

AI-assisted analysis of shareAI-lab/learn-claude-code@985456f4ad (2026-08-14). Data as JSON: /api/errors/0a9c16d6cd116507. Report an issue: GitHub.