Yeachan-Heo/oh-my-codex · error · Error

sandbox.md frontmatter must define an evaluator block.

Error message

sandbox.md frontmatter must define an evaluator block.

What it means

parseSandboxContract throws EVALUATOR_BLOCK_ERROR when the parsed frontmatter has no 'evaluator' key, or evaluator is not a plain object (it is a string/array). The evaluator block is the core of the autoresearch v1 sandbox contract and must be a mapping containing command and format.

Source

Thrown at src/autoresearch/contracts.ts:153

function parseKeepPolicy(raw: unknown): AutoresearchKeepPolicy | undefined {
  if (raw === undefined) return undefined;
  if (typeof raw !== 'string') {
    throw contractError('sandbox.md frontmatter evaluator.keep_policy must be a string when provided.');
  }
  const normalized = raw.trim().toLowerCase();
  if (!normalized) return undefined;
  if (normalized === 'pass_only') return 'pass_only';
  if (normalized === 'score_improvement') return 'score_improvement';
  throw contractError('sandbox.md frontmatter evaluator.keep_policy must be one of: score_improvement, pass_only.');
}

export function parseSandboxContract(content: string): ParsedSandboxContract {
  const { frontmatter, body } = extractFrontmatter(content);
  const parsedFrontmatter = parseSimpleYamlFrontmatter(frontmatter);
  const evaluatorRaw = parsedFrontmatter.evaluator;

  if (!evaluatorRaw || typeof evaluatorRaw !== 'object' || Array.isArray(evaluatorRaw)) {
    throw contractError(EVALUATOR_BLOCK_ERROR);
  }

  const evaluator = evaluatorRaw as { command?: unknown; format?: unknown; keep_policy?: unknown };
  const command = typeof evaluator.command === 'string'
    ? evaluator.command.trim()
    : '';
  const format = typeof evaluator.format === 'string'
    ? evaluator.format.trim().toLowerCase()
    : '';
  const keepPolicy = parseKeepPolicy(evaluator.keep_policy);

  if (!command) {
    throw contractError(EVALUATOR_COMMAND_ERROR);
  }
  if (!format) {
    throw contractError(EVALUATOR_FORMAT_REQUIRED_ERROR);
  }
  if (format !== 'json') {

View on GitHub (pinned to 3ad79a8a6f)

Solutions

1. Add an 'evaluator:' section header with nested 'command:' and 'format: json' lines.
2. Ensure evaluator is a mapping, not an inline scalar or list.
3. Validate sandbox.md with parseSandboxContract in a test or preflight script before running the mission.

Example fix

# before
command: ./run.sh
format: json

# after
evaluator:
  command: ./run.sh
  format: json

function hasEvaluatorBlock(content: string): boolean {
  const fm = content.match(/^---\r?\n([\s\S]*?)\r?\n---/);
  if (!fm) return false;
  return /^[A-Za-z0-9_-]+:\s*$/m.test(fm[1]) && /(^|\n)evaluator:\s*(\n|$)/.test(fm[1]);
}

const hasEvaluatorObject = (fm: Record<string, unknown>): boolean =>
  !!fm.evaluator && typeof fm.evaluator === 'object' && !Array.isArray(fm.evaluator);

try {
  parseSandboxContract(content);
} catch (err) {
  if ((err as Error).message.includes('must define an evaluator block')) {
    // add an 'evaluator:' section with command and format: json
  }
  throw err;
}

When it happens

Trigger: sandbox.md frontmatter lacking an 'evaluator:' section; 'evaluator: ./run.sh' (scalar); or evaluator defined as a list. Occurs wherever parseSandboxContract is used: loading contracts, writing draft/deep-interview artifacts, parsing draft artifacts, and reading persisted results.

Common situations: Forgetting the evaluator block when authoring sandbox.md; inlining the command at the top level instead of under evaluator; malformed nesting that makes the parser store a scalar.

Related errors

• sandbox.md frontmatter evaluator.command is required.
• sandbox.md frontmatter evaluator.format is required and must
• sandbox.md frontmatter evaluator.format must be json in auto
• ${filePath} frontmatter "${key}" must not be empty
• ${filePath} frontmatter "${key}" must be a single-line strin

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/bfd1573e55bef3d3. Report an issue: GitHub.