Yeachan-Heo/oh-my-codex · error · Error

Evaluator output must include boolean pass.

Error message

Evaluator output must include boolean pass.

What it means

parseEvaluatorResult throws when the parsed JSON object lacks a 'pass' field of boolean type. pass is the mandatory verdict of the evaluator; without it the result cannot be interpreted as success or failure.

Source

Thrown at src/autoresearch/contracts.ts:200

    body,
  };
}

export function parseEvaluatorResult(raw: string): AutoresearchEvaluatorResult {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    throw contractError('Evaluator output must be valid JSON with required boolean pass and optional numeric score.');
  }

  if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
    throw contractError('Evaluator output must be a JSON object.');
  }

  const result = parsed as Record<string, unknown>;
  if (typeof result.pass !== 'boolean') {
    throw contractError('Evaluator output must include boolean pass.');
  }
  if (result.score !== undefined && typeof result.score !== 'number') {
    throw contractError('Evaluator output score must be numeric when provided.');
  }

  return {
    pass: result.pass,
    ...(result.score === undefined ? {} : { score: result.score }),
  };
}

export async function loadAutoresearchMissionContract(missionDirArg: string): Promise<AutoresearchMissionContract> {
  const missionDir = resolve(missionDirArg);
  if (!existsSync(missionDir)) {
    throw contractError(`mission-dir does not exist: ${missionDir}`);
  }

  const repoRoot = readGit(missionDir, ['rev-parse', '--show-toplevel']);

View on GitHub (pinned to 3ad79a8a6f)

Solutions

1. Include a literal boolean pass field: '{"pass": true, "score": 0.9}'.
2. Use JSON.stringify({pass, score}) in the evaluator rather than string templates.
3. Check for exact key name 'pass' and boolean type (no quotes).

Example fix

# before
echo '{"passed":true,"score":0.9}'

# after
echo '{"pass":true,"score":0.9}'

function outputHasBooleanPass(raw: string): boolean {
  try {
    const v = JSON.parse(raw);
    return !!v && typeof v === 'object' && !Array.isArray(v)
      && typeof (v as any).pass === 'boolean';
  } catch { return false; }
}

const hasBooleanPass = (v: unknown): boolean =>
  isJsonObject(v) && typeof (v as Record<string, unknown>).pass === 'boolean';

try {
  parseEvaluatorResult(stdout);
} catch (err) {
  if ((err as Error).message.includes('boolean pass')) {
    // rename passed/success -> pass and ensure it is a real boolean
  }
  throw err;
}

When it happens

Trigger: Evaluator prints '{"score": 0.9}' (no pass), '{"passed": true}' (wrong key name), or '{"pass": "true"}' (string instead of boolean).

Common situations: Key naming mismatches (passed/success/ok vs pass); truthy strings from shell-built JSON; forgetting to add pass when adding score reporting.

Related errors

• sandbox.md frontmatter evaluator.command is required.
• sandbox.md frontmatter evaluator.format is required and must
• Evaluator output must be valid JSON with required boolean pa
• Evaluator output must be a JSON object.
• Evaluator output score must be numeric when provided.

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/b3179db2dcb4ce14. Report an issue: GitHub.