{"record":{"id":"a5bfb0e52b363b88","repo":"Yeachan-Heo/oh-my-codex","slug":"evaluator-output-must-be-valid-json-with-required","errorCode":null,"errorMessage":"Evaluator output must be valid JSON with required boolean pass and optional numeric score.","messagePattern":"Evaluator output must be valid JSON with required boolean pass and optional numeric score\\.","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"src/autoresearch/contracts.ts","lineNumber":191,"sourceCode":"  }\n\n  return {\n    frontmatter: parsedFrontmatter,\n    evaluator: {\n      command,\n      format: 'json',\n      ...(keepPolicy ? { keep_policy: keepPolicy } : {}),\n    },\n    body,\n  };\n}\n\nexport function parseEvaluatorResult(raw: string): AutoresearchEvaluatorResult {\n  let parsed: unknown;\n  try {\n    parsed = JSON.parse(raw);\n  } catch {\n    throw contractError('Evaluator output must be valid JSON with required boolean pass and optional numeric score.');\n  }\n\n  if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {\n    throw contractError('Evaluator output must be a JSON object.');\n  }\n\n  const result = parsed as Record<string, unknown>;\n  if (typeof result.pass !== 'boolean') {\n    throw contractError('Evaluator output must include boolean pass.');\n  }\n  if (result.score !== undefined && typeof result.score !== 'number') {\n    throw contractError('Evaluator output score must be numeric when provided.');\n  }\n\n  return {\n    pass: result.pass,\n    ...(result.score === undefined ? {} : { score: result.score }),\n  };","sourceCodeStart":173,"sourceCodeEnd":209,"githubUrl":"https://github.com/Yeachan-Heo/oh-my-codex/blob/3ad79a8a6fe6e95fdbb8c00e40716fffe4011ce2/src/autoresearch/contracts.ts#L173-L209","documentation":"parseEvaluatorResult throws when the raw string captured from the evaluator command's output cannot be JSON.parse'd. The contract requires the evaluator to print a JSON object with a boolean pass (and optional numeric score); malformed JSON such as partial output, leading log lines, or trailing commas fails here.","triggerScenarios":"Evaluator command prints 'pass' instead of '{\\\"pass\\\":true}', mixes logs before/after the JSON, truncates output, or emits NaN/Infinity which JSON.parse rejects.","commonSituations":"Evaluator scripts that log progress to stdout and only intend the last line as the result; NaN scores serialized via JSON.stringify (which yields null-safe but 'NaN' literal breaks parse if hand-built); truncated output from killed processes.","solutions":["Make the evaluator print exactly one JSON object to stdout; route logs to stderr.","Validate output in the evaluator itself (JSON.stringify an object) before exiting.","If wrapping a legacy evaluator, capture its text result and have the wrapper print {\\\"pass\\\": ..., \\\"score\\\": ...}."],"exampleFix":"# before (evaluate.sh)\necho \"running...\"\necho \"pass\"\n\n# after (evaluate.sh)\necho \"running...\" >&2\necho '{\"pass\":true,\"score\":1}'","handlingStrategy":"validation","validationCode":"import { parseEvaluatorResult } from './contracts';\n\nfunction evaluatorOutputLooksLikeJson(raw: string): boolean {\n  try { JSON.parse(raw); return true; } catch { return false; }\n}\n\n// Before trusting output, dry-run the evaluator command and check:\n//   child stdout parses as JSON","typeGuard":"const isParsableJson = (raw: string): boolean => { try { JSON.parse(raw); return true; } catch { return false; } };","tryCatchPattern":"try {\n  const result = parseEvaluatorResult(stdout);\n} catch (err) {\n  if ((err as Error).message.includes('must be valid JSON')) {\n    // inspect evaluator stdout; strip log lines or fix serialization and re-run\n  }\n  throw err;\n}","preventionTips":["Evaluators must print exactly one JSON object to stdout.","Send any logging to stderr.","Test evaluators standalone: node -e 'JSON.parse(require(\"fs\").readFileSync(0,\"utf8\"))'."],"tags":["evaluator","json","parsing","contract-validation"],"backgroundTag":"invalid-json-output","analyzedSha":"3ad79a8a6fe6e95fdbb8c00e40716fffe4011ce2","analyzedAt":"2026-08-27T22:18:39.783Z","schemaVersion":2},"datasetVersion":"2026-08-28T00:17:15.603Z"}