Yeachan-Heo/oh-my-codex · error · Error

Evaluator output must be valid JSON with required boolean pa

Error message

Evaluator output must be valid JSON with required boolean pass and optional numeric score.

What it means

parseEvaluatorResult throws when the raw string captured from the evaluator command's output cannot be JSON.parse'd. The contract requires the evaluator to print a JSON object with a boolean pass (and optional numeric score); malformed JSON such as partial output, leading log lines, or trailing commas fails here.

Source

Thrown at src/autoresearch/contracts.ts:191

  }

  return {
    frontmatter: parsedFrontmatter,
    evaluator: {
      command,
      format: 'json',
      ...(keepPolicy ? { keep_policy: keepPolicy } : {}),
    },
    body,
  };
}

export function parseEvaluatorResult(raw: string): AutoresearchEvaluatorResult {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    throw contractError('Evaluator output must be valid JSON with required boolean pass and optional numeric score.');
  }

  if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
    throw contractError('Evaluator output must be a JSON object.');
  }

  const result = parsed as Record<string, unknown>;
  if (typeof result.pass !== 'boolean') {
    throw contractError('Evaluator output must include boolean pass.');
  }
  if (result.score !== undefined && typeof result.score !== 'number') {
    throw contractError('Evaluator output score must be numeric when provided.');
  }

  return {
    pass: result.pass,
    ...(result.score === undefined ? {} : { score: result.score }),
  };

View on GitHub (pinned to 3ad79a8a6f)

Solutions

1. Make the evaluator print exactly one JSON object to stdout; route logs to stderr.
2. Validate output in the evaluator itself (JSON.stringify an object) before exiting.
3. If wrapping a legacy evaluator, capture its text result and have the wrapper print {\"pass\": ..., \"score\": ...}.

Example fix

# before (evaluate.sh)
echo "running..."
echo "pass"

# after (evaluate.sh)
echo "running..." >&2
echo '{"pass":true,"score":1}'

import { parseEvaluatorResult } from './contracts';

function evaluatorOutputLooksLikeJson(raw: string): boolean {
  try { JSON.parse(raw); return true; } catch { return false; }
}

// Before trusting output, dry-run the evaluator command and check:
//   child stdout parses as JSON

const isParsableJson = (raw: string): boolean => { try { JSON.parse(raw); return true; } catch { return false; } };

try {
  const result = parseEvaluatorResult(stdout);
} catch (err) {
  if ((err as Error).message.includes('must be valid JSON')) {
    // inspect evaluator stdout; strip log lines or fix serialization and re-run
  }
  throw err;
}

When it happens

Trigger: Evaluator command prints 'pass' instead of '{\"pass\":true}', mixes logs before/after the JSON, truncates output, or emits NaN/Infinity which JSON.parse rejects.

Common situations: Evaluator scripts that log progress to stdout and only intend the last line as the result; NaN scores serialized via JSON.stringify (which yields null-safe but 'NaN' literal breaks parse if hand-built); truncated output from killed processes.

Related errors

• autoresearch candidate artifact must be valid JSON
• ${source} must be valid JSON: ${(error as Error).message}
• visual_verdict.${field} must be an array
• sandbox.md frontmatter must define an evaluator block.
• Evaluator output must be a JSON object.

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/a5bfb0e52b363b88. Report an issue: GitHub.