linshenkx/prompt-optimizer · error · EvaluationValidationError

Compare evaluation requires at least two snapshots.

Error message

Compare evaluation requires at least two snapshots.

What it means

Thrown for type:'compare' when request.snapshots is missing, not an array, or has fewer than two entries. A comparison needs at least two candidate snapshots to rank against each other, so the service rejects the request during validateRequest with an EvaluationValidationError.

Source

Thrown at packages/core/src/services/evaluation/service.ts:494

            'Result evaluation snapshot testCaseId must match testCase.id.'
          );
        }
        if (this.isImageText2ImageMode(request) && !this.hasSnapshotOutputMedia(request.snapshot)) {
          throw new EvaluationValidationError(
            'Image result evaluation requires at least one output image evidence item.'
          );
        }
        break;

      case 'compare':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        if (!Array.isArray(request.testCases) || request.testCases.length < 1) {
          throw new EvaluationValidationError('Compare evaluation requires at least one test case.');
        }
        if (!Array.isArray(request.snapshots) || request.snapshots.length < 2) {
          throw new EvaluationValidationError('Compare evaluation requires at least two snapshots.');
        }

        const testCaseIds = new Set<string>();
        request.testCases.forEach((testCase, index) => {
          this.validateTestCase(testCase, `Compare test case #${index + 1}`);
          if (testCaseIds.has(testCase.id)) {
            throw new EvaluationValidationError(
              `Compare test case #${index + 1} id must be unique.`
            );
          }
          testCaseIds.add(testCase.id);
        });

        request.snapshots.forEach((snapshot, index) => {
          this.validateSnapshot(snapshot, `Compare snapshot #${index + 1}`);
          if (!testCaseIds.has(snapshot.testCaseId)) {
            throw new EvaluationValidationError(
              `Compare snapshot #${index + 1} references unknown testCaseId "${snapshot.testCaseId}".`

View on GitHub (pinned to 3e677b1d9f)

Solutions

1. Provide at least two snapshots (one per variant/candidate) before comparing
2. Wait for all variant runs to complete and capture their snapshots before enabling compare
3. If only one snapshot exists, run a 'result' evaluation instead

Example fix

// before
await svc.evaluate({ type: 'compare', testCases, snapshots: [snapA], ... });

// after
await svc.evaluate({ type: 'compare', testCases, snapshots: [snapA, snapB], ... });

if (!Array.isArray(req.snapshots) || req.snapshots.length < 2) throw new Error('Need >=2 snapshots to compare');
await svc.evaluate(req);

const comparable = (r: CompareRequest) => Array.isArray(r.snapshots) && r.snapshots.length >= 2;

When it happens

Trigger: evaluate({type:'compare', snapshots: []}) or snapshots with only one element, or snapshots not an array.

Common situations: Only one variant finished generating so the second snapshot is missing; snapshot list built from a single selected run; race where the second snapshot is still being captured when compare fires.

Related errors

• Compare snapshot #${index + 1} references unknown testCaseId
• Result evaluation snapshot testCaseId must match testCase.id
• Compare evaluation requires at least one test case.
• Compare test case #${index + 1} id must be unique.
• Image compare evaluation requires at least two snapshots wit

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/73fc3bba8eeeeb16. Report an issue: GitHub.