Yeachan-Heo/oh-my-codex · error · Error

Evaluator output score must be numeric when provided.

Error message

Evaluator output score must be numeric when provided.

What it means

parseEvaluatorResult throws when the output object includes a 'score' field whose value is not a number. score is optional, but when present it must be numeric so downstream ranking/keep-policy logic can compare it.

Source

Thrown at src/autoresearch/contracts.ts:203

export function parseEvaluatorResult(raw: string): AutoresearchEvaluatorResult {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    throw contractError('Evaluator output must be valid JSON with required boolean pass and optional numeric score.');
  }

  if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
    throw contractError('Evaluator output must be a JSON object.');
  }

  const result = parsed as Record<string, unknown>;
  if (typeof result.pass !== 'boolean') {
    throw contractError('Evaluator output must include boolean pass.');
  }
  if (result.score !== undefined && typeof result.score !== 'number') {
    throw contractError('Evaluator output score must be numeric when provided.');
  }

  return {
    pass: result.pass,
    ...(result.score === undefined ? {} : { score: result.score }),
  };
}

export async function loadAutoresearchMissionContract(missionDirArg: string): Promise<AutoresearchMissionContract> {
  const missionDir = resolve(missionDirArg);
  if (!existsSync(missionDir)) {
    throw contractError(`mission-dir does not exist: ${missionDir}`);
  }

  const repoRoot = readGit(missionDir, ['rev-parse', '--show-toplevel']);
  ensurePathInside(repoRoot, missionDir);

  const missionFile = join(missionDir, 'mission.md');

View on GitHub (pinned to 3ad79a8a6f)

Solutions

1. Emit score as a bare JSON number or omit the key entirely.
2. Use JSON.stringify so numbers stay numbers.
3. If score is unavailable, leave the field out rather than sending null or ''.

Example fix

# before
echo '{"pass":true,"score":"0.9"}'

# after
echo '{"pass":true,"score":0.9}'

function outputScoreIsNumericIfPresent(raw: string): boolean {
  try {
    const v = JSON.parse(raw) as Record<string, unknown>;
    return v.score === undefined || typeof v.score === 'number';
  } catch { return false; }
}

const hasNumericScore = (v: Record<string, unknown>): boolean =>
  v.score === undefined || typeof v.score === 'number';

try {
  parseEvaluatorResult(stdout);
} catch (err) {
  if ((err as Error).message.includes('score must be numeric')) {
    // emit score as a JSON number or omit the field
  }
  throw err;
}

When it happens

Trigger: Evaluator prints '{"pass":true,"score":"0.9"}' (string), score: null, or score: true — typeof !== 'number' fails the check.

Common situations: Scores serialized as strings from shell or CSV-derived pipelines; null used as 'no score' instead of omitting the key; booleans from flag-style scripts.

Related errors

• ${source} must decode to a JSON object
• ${label} must be a JSON object.
• Evaluator output must be valid JSON with required boolean pa
• Evaluator output must be a JSON object.
• Evaluator output must include boolean pass.

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/c716ba87674ca689. Report an issue: GitHub.