Yeachan-Heo/oh-my-codex · error · Error

sandbox.md frontmatter evaluator.format must be json in…

Error message

sandbox.md frontmatter evaluator.format must be json in autoresearch v1.

What it means

parseSandboxContract throws EVALUATOR_FORMAT_JSON_ERROR when evaluator.format is present but its lowercased/trimmed value is not exactly 'json'. Autoresearch v1 only supports JSON evaluator output, so any other format (text, table, csv, json5) is rejected.

Solutions

1. Change the evaluator to emit JSON and set 'format: json'.
2. If your evaluator prints text, wrap/modify it to print a JSON object with boolean pass and optional numeric score, then declare json.

Example fix

# before
evaluator:
  command: ./evaluate.sh --pretty
  format: text

# after
evaluator:
  command: ./evaluate.sh --json
  format: json

function evaluatorFormatIsJson(content: string): boolean {
  const m = /^[ \t]+format:\s*(\S+)/m.exec(content);
  return !!m && m[1].trim().toLowerCase() === 'json';
}

const isJsonFormat = (ev: { format?: unknown }): boolean =>
  typeof ev.format === 'string' && ev.format.trim().toLowerCase() === 'json';

try {
  parseSandboxContract(content);
} catch (err) {
  if ((err as Error).message.includes('format must be json')) {
    // switch evaluator output to a JSON object and set format: json
  }
  throw err;
}

When it happens

Trigger: Frontmatter with 'format: text', 'format: csv', 'format: JSON5', or a typo like 'jason' after normalization.

Common situations: Porting a sandbox from another tool that emitted human-readable text; typos; attempting structured-but-not-JSON formats.

Related errors

• sandbox.md frontmatter evaluator.command is required.
• sandbox.md frontmatter evaluator.format is required and…
• sandbox.md frontmatter must define an evaluator block.
• frontmatter " " has an unterminated quoted string
• frontmatter " " must be a single-line string

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/2b809f2a1653254f. Report an issue: GitHub.

Appendix: source

Thrown at src/autoresearch/contracts.ts:172

  }

  const evaluator = evaluatorRaw as { command?: unknown; format?: unknown; keep_policy?: unknown };
  const command = typeof evaluator.command === 'string'
    ? evaluator.command.trim()
    : '';
  const format = typeof evaluator.format === 'string'
    ? evaluator.format.trim().toLowerCase()
    : '';
  const keepPolicy = parseKeepPolicy(evaluator.keep_policy);

  if (!command) {
    throw contractError(EVALUATOR_COMMAND_ERROR);
  }
  if (!format) {
    throw contractError(EVALUATOR_FORMAT_REQUIRED_ERROR);
  }
  if (format !== 'json') {
    throw contractError(EVALUATOR_FORMAT_JSON_ERROR);
  }

  return {
    frontmatter: parsedFrontmatter,
    evaluator: {
      command,
      format: 'json',
      ...(keepPolicy ? { keep_policy: keepPolicy } : {}),
    },
    body,
  };
}

export function parseEvaluatorResult(raw: string): AutoresearchEvaluatorResult {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {

View on GitHub (pinned to 3ad79a8a6f)