Yeachan-Heo/oh-my-codex · error · Error

sandbox.md frontmatter evaluator.format must be json in auto

Error message

sandbox.md frontmatter evaluator.format must be json in autoresearch v1.

What it means

parseSandboxContract throws EVALUATOR_FORMAT_JSON_ERROR when evaluator.format is present but its lowercased/trimmed value is not exactly 'json'. Autoresearch v1 only supports JSON evaluator output, so any other format (text, table, csv, json5) is rejected.

Source

Thrown at src/autoresearch/contracts.ts:172

  }

  const evaluator = evaluatorRaw as { command?: unknown; format?: unknown; keep_policy?: unknown };
  const command = typeof evaluator.command === 'string'
    ? evaluator.command.trim()
    : '';
  const format = typeof evaluator.format === 'string'
    ? evaluator.format.trim().toLowerCase()
    : '';
  const keepPolicy = parseKeepPolicy(evaluator.keep_policy);

  if (!command) {
    throw contractError(EVALUATOR_COMMAND_ERROR);
  }
  if (!format) {
    throw contractError(EVALUATOR_FORMAT_REQUIRED_ERROR);
  }
  if (format !== 'json') {
    throw contractError(EVALUATOR_FORMAT_JSON_ERROR);
  }

  return {
    frontmatter: parsedFrontmatter,
    evaluator: {
      command,
      format: 'json',
      ...(keepPolicy ? { keep_policy: keepPolicy } : {}),
    },
    body,
  };
}

export function parseEvaluatorResult(raw: string): AutoresearchEvaluatorResult {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {

View on GitHub (pinned to 3ad79a8a6f)

Solutions

1. Change the evaluator to emit JSON and set 'format: json'.
2. If your evaluator prints text, wrap/modify it to print a JSON object with boolean pass and optional numeric score, then declare json.

Example fix

# before
evaluator:
  command: ./evaluate.sh --pretty
  format: text

# after
evaluator:
  command: ./evaluate.sh --json
  format: json

function evaluatorFormatIsJson(content: string): boolean {
  const m = /^[ \t]+format:\s*(\S+)/m.exec(content);
  return !!m && m[1].trim().toLowerCase() === 'json';
}

const isJsonFormat = (ev: { format?: unknown }): boolean =>
  typeof ev.format === 'string' && ev.format.trim().toLowerCase() === 'json';

try {
  parseSandboxContract(content);
} catch (err) {
  if ((err as Error).message.includes('format must be json')) {
    // switch evaluator output to a JSON object and set format: json
  }
  throw err;
}

When it happens

Trigger: Frontmatter with 'format: text', 'format: csv', 'format: JSON5', or a typo like 'jason' after normalization.

Common situations: Porting a sandbox from another tool that emitted human-readable text; typos; attempting structured-but-not-JSON formats.

Related errors

• sandbox.md frontmatter must define an evaluator block.
• sandbox.md frontmatter evaluator.command is required.
• sandbox.md frontmatter evaluator.format is required and must
• ${filePath} frontmatter "${key}" must not be empty
• ${filePath} frontmatter "${key}" must be a single-line strin

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/2b809f2a1653254f. Report an issue: GitHub.