linshenkx/prompt-optimizer · error · EvaluationValidationError

Compare snapshot #${index + 1} references unknown testCaseId

Error message

Compare snapshot #${index + 1} references unknown testCaseId "${snapshot.testCaseId}".

What it means

Thrown during compare validation when a snapshot's testCaseId does not appear among the ids of the provided testCases array. Every compared snapshot must belong to a known test case so the evaluator can group candidates per case. The message includes the 1-based snapshot index and the offending testCaseId.

Source

Thrown at packages/core/src/services/evaluation/service.ts:511

        if (!Array.isArray(request.snapshots) || request.snapshots.length < 2) {
          throw new EvaluationValidationError('Compare evaluation requires at least two snapshots.');
        }

        const testCaseIds = new Set<string>();
        request.testCases.forEach((testCase, index) => {
          this.validateTestCase(testCase, `Compare test case #${index + 1}`);
          if (testCaseIds.has(testCase.id)) {
            throw new EvaluationValidationError(
              `Compare test case #${index + 1} id must be unique.`
            );
          }
          testCaseIds.add(testCase.id);
        });

        request.snapshots.forEach((snapshot, index) => {
          this.validateSnapshot(snapshot, `Compare snapshot #${index + 1}`);
          if (!testCaseIds.has(snapshot.testCaseId)) {
            throw new EvaluationValidationError(
              `Compare snapshot #${index + 1} references unknown testCaseId "${snapshot.testCaseId}".`
            );
          }
        });
        if (this.isImageText2ImageMode(request)) {
          const snapshotsWithMedia = request.snapshots.filter((snapshot) =>
            this.hasSnapshotOutputMedia(snapshot)
          );
          if (snapshotsWithMedia.length < 2) {
            throw new EvaluationValidationError(
              'Image compare evaluation requires at least two snapshots with output image evidence.'
            );
          }
          if (snapshotsWithMedia.length !== request.snapshots.length) {
            throw new EvaluationValidationError(
              'Image compare evaluation requires every compared snapshot to include output image evidence.'
            );
          }

View on GitHub (pinned to 3e677b1d9f)

Solutions

1. Filter snapshots to those whose testCaseId exists in the testCases list before calling evaluate
2. Refresh/re-capture snapshots after test case ids change
3. Include the missing test case in request.testCases if it should participate

Example fix

// before
await svc.evaluate({ type: 'compare', testCases, snapshots: allSnapshots, ... });

// after
const ids = new Set(testCases.map(tc => tc.id));
const snapshots = allSnapshots.filter(s => ids.has(s.testCaseId));
await svc.evaluate({ type: 'compare', testCases, snapshots, ... });

const ids = new Set(req.testCases.map(tc => tc.id));
const orphans = req.snapshots.filter(s => !ids.has(s.testCaseId));
if (orphans.length) throw new Error(`Orphan snapshots: ${orphans.map(s => s.testCaseId).join(', ')}`);
await svc.evaluate(req);

const snapshotsAllReferenced = (snaps: EvaluationSnapshot[], tcs: EvaluationTestCase[]) =>
  snaps.every(s => new Set(tcs.map(t => t.id)).has(s.testCaseId));

When it happens

Trigger: evaluate({type:'compare'}) where some snapshot.testCaseId is not in the set of testCases[].id — e.g. snapshot from a deleted test case, typo'd id, or stale fixture data.

Common situations: Test cases deleted or re-keyed after snapshots were captured; snapshots loaded from another workspace/run; frontend filtering test cases but not their snapshots.

Related errors

• Compare evaluation requires at least two snapshots.
• Result evaluation snapshot testCaseId must match testCase.id
• Compare evaluation requires at least one test case.
• Compare test case #${index + 1} id must be unique.
• Image compare evaluation requires at least two snapshots wit

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/f31812a996d6be67. Report an issue: GitHub.