linshenkx/prompt-optimizer · error · EvaluationValidationError

Compare evaluation requires at least two snapshots.

Error message

Compare evaluation requires at least two snapshots.

What it means

Thrown for type:'compare' when request.snapshots is missing, not an array, or has fewer than two entries. A comparison needs at least two candidate snapshots to rank against each other, so the service rejects the request during validateRequest with an EvaluationValidationError.

Solutions

  1. Provide at least two snapshots (one per variant/candidate) before comparing
  2. Wait for all variant runs to complete and capture their snapshots before enabling compare
  3. If only one snapshot exists, run a 'result' evaluation instead

Example fix

// before
await svc.evaluate({ type: 'compare', testCases, snapshots: [snapA], ... });

// after
await svc.evaluate({ type: 'compare', testCases, snapshots: [snapA, snapB], ... });
Defensive patterns

Strategy: validation

Validate before calling

if (!Array.isArray(req.snapshots) || req.snapshots.length < 2) throw new Error('Need >=2 snapshots to compare');
await svc.evaluate(req);

Type guard

const comparable = (r: CompareRequest) => Array.isArray(r.snapshots) && r.snapshots.length >= 2;

Prevention

When it happens

Trigger: evaluate({type:'compare', snapshots: []}) or snapshots with only one element, or snapshots not an array.

Common situations: Only one variant finished generating so the second snapshot is missing; snapshot list built from a single selected run; race where the second snapshot is still being captured when compare fires.

Related errors

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/73fc3bba8eeeeb16. Report an issue: GitHub.

Appendix: source

Thrown at packages/core/src/services/evaluation/service.ts:494

            'Result evaluation snapshot testCaseId must match testCase.id.'
          );
        }
        if (this.isImageText2ImageMode(request) && !this.hasSnapshotOutputMedia(request.snapshot)) {
          throw new EvaluationValidationError(
            'Image result evaluation requires at least one output image evidence item.'
          );
        }
        break;

      case 'compare':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        if (!Array.isArray(request.testCases) || request.testCases.length < 1) {
          throw new EvaluationValidationError('Compare evaluation requires at least one test case.');
        }
        if (!Array.isArray(request.snapshots) || request.snapshots.length < 2) {
          throw new EvaluationValidationError('Compare evaluation requires at least two snapshots.');
        }

        const testCaseIds = new Set<string>();
        request.testCases.forEach((testCase, index) => {
          this.validateTestCase(testCase, `Compare test case #${index + 1}`);
          if (testCaseIds.has(testCase.id)) {
            throw new EvaluationValidationError(
              `Compare test case #${index + 1} id must be unique.`
            );
          }
          testCaseIds.add(testCase.id);
        });

        request.snapshots.forEach((snapshot, index) => {
          this.validateSnapshot(snapshot, `Compare snapshot #${index + 1}`);
          if (!testCaseIds.has(snapshot.testCaseId)) {
            throw new EvaluationValidationError(
              `Compare snapshot #${index + 1} references unknown testCaseId "${snapshot.testCaseId}".`

View on GitHub (pinned to 3e677b1d9f)