linshenkx/prompt-optimizer · error · EvaluationValidationError

Compare evaluation requires at least one test case.

Error message

Compare evaluation requires at least one test case.

What it means

Thrown for type:'compare' when request.testCases is not an array or has zero entries. Compare evaluation judges multiple snapshots grouped by test case, so at least one test case is required. It is an EvaluationValidationError raised in validateRequest before execution.

Source

Thrown at packages/core/src/services/evaluation/service.ts:491

        this.validateSnapshot(request.snapshot, 'Result evaluation snapshot');
        if (request.snapshot.testCaseId !== request.testCase.id) {
          throw new EvaluationValidationError(
            'Result evaluation snapshot testCaseId must match testCase.id.'
          );
        }
        if (this.isImageText2ImageMode(request) && !this.hasSnapshotOutputMedia(request.snapshot)) {
          throw new EvaluationValidationError(
            'Image result evaluation requires at least one output image evidence item.'
          );
        }
        break;

      case 'compare':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        if (!Array.isArray(request.testCases) || request.testCases.length < 1) {
          throw new EvaluationValidationError('Compare evaluation requires at least one test case.');
        }
        if (!Array.isArray(request.snapshots) || request.snapshots.length < 2) {
          throw new EvaluationValidationError('Compare evaluation requires at least two snapshots.');
        }

        const testCaseIds = new Set<string>();
        request.testCases.forEach((testCase, index) => {
          this.validateTestCase(testCase, `Compare test case #${index + 1}`);
          if (testCaseIds.has(testCase.id)) {
            throw new EvaluationValidationError(
              `Compare test case #${index + 1} id must be unique.`
            );
          }
          testCaseIds.add(testCase.id);
        });

        request.snapshots.forEach((snapshot, index) => {
          this.validateSnapshot(snapshot, `Compare snapshot #${index + 1}`);

View on GitHub (pinned to 3e677b1d9f)

Solutions

1. Populate request.testCases with at least one valid test case before calling evaluate
2. If you only have one test case and one snapshot, use type:'result' instead of 'compare'
3. Guard the compare action in the UI until a test case is selected

Example fix

// before
await svc.evaluate({ type: 'compare', testCases: [], snapshots, ... });

// after
await svc.evaluate({ type: 'compare', testCases: [tc], snapshots, ... });

if (!Array.isArray(req.testCases) || req.testCases.length < 1) throw new Error('Select at least one test case to compare');
await svc.evaluate(req);

const hasTestCases = (r: unknown): r is CompareRequest =>
  Array.isArray((r as CompareRequest).testCases) && (r as CompareRequest).testCases!.length >= 1;

When it happens

Trigger: evaluate({type:'compare'}) with testCases omitted, null, set to [], or not an array (e.g. an object map of test cases).

Common situations: Passing a singular testCase (the 'result' shape) to a compare request; filter that removed all test cases (none selected in UI); test cases loaded asynchronously and still empty when evaluate fired.

Related errors

• Result evaluation snapshot testCaseId must match testCase.id
• Compare evaluation requires at least two snapshots.
• Compare test case #${index + 1} id must be unique.
• Compare snapshot #${index + 1} references unknown testCaseId
• Image compare evaluation requires at least two snapshots wit

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/5dfd9626d92cd7eb. Report an issue: GitHub.