linshenkx/prompt-optimizer · error · EvaluationValidationError

Image compare evaluation requires at least two snapshots wit

Error message

Image compare evaluation requires at least two snapshots with output image evidence.

What it means

Thrown for compare evaluations in image text-2-image mode when fewer than two snapshots contain output image evidence. Image comparison needs at least two images to compare, so validateRequest rejects the request. Counted via hasSnapshotOutputMedia on each snapshot.

Source

Thrown at packages/core/src/services/evaluation/service.ts:521

            );
          }
          testCaseIds.add(testCase.id);
        });

        request.snapshots.forEach((snapshot, index) => {
          this.validateSnapshot(snapshot, `Compare snapshot #${index + 1}`);
          if (!testCaseIds.has(snapshot.testCaseId)) {
            throw new EvaluationValidationError(
              `Compare snapshot #${index + 1} references unknown testCaseId "${snapshot.testCaseId}".`
            );
          }
        });
        if (this.isImageText2ImageMode(request)) {
          const snapshotsWithMedia = request.snapshots.filter((snapshot) =>
            this.hasSnapshotOutputMedia(snapshot)
          );
          if (snapshotsWithMedia.length < 2) {
            throw new EvaluationValidationError(
              'Image compare evaluation requires at least two snapshots with output image evidence.'
            );
          }
          if (snapshotsWithMedia.length !== request.snapshots.length) {
            throw new EvaluationValidationError(
              'Image compare evaluation requires every compared snapshot to include output image evidence.'
            );
          }
        }
        break;

      case 'prompt-only':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        break;

      case 'prompt-iterate':

View on GitHub (pinned to 3e677b1d9f)

Solutions

1. Ensure each compared variant snapshot includes output image evidence (assetId or b64)
2. Exclude snapshots lacking images from the compare set (then re-check the >=2 requirement)
3. Re-run the failed image generation steps before comparing

Example fix

// before
await svc.evaluate({ type: 'compare', mode, snapshots: [withImg, textOnly], ... });

// after
const snaps = [snapA, snapB].map(s => ({ ...s, outputEvidence: [{ assetId: s.imageId }] }));
await svc.evaluate({ type: 'compare', mode, snapshots: snaps, ... });

const withMedia = snapshots.filter(hasSnapshotOutputMedia);
if (isImageMode(mode) && withMedia.length < 2) throw new Error('Need >=2 snapshots with output images');
await svc.evaluate({ ...req, snapshots: withMedia });

const hasSnapshotOutputMedia = (s: EvaluationSnapshot) =>
  (s.outputEvidence ?? []).some(e => Boolean(e.assetId?.trim() || e.b64?.trim()));

When it happens

Trigger: evaluate({type:'compare', mode:<image text-2-image>, snapshots}) where 0 or 1 snapshots have output image media.

Common situations: Some variant runs finished text-only or failed before generating an image; snapshot evidence media lost in serialization; mixing text and image variants in one compare request in image mode.

Related errors

• Image compare evaluation requires every compared snapshot to
• Image result evaluation requires at least one output image e
• Evaluation image evidence "${label}" is missing both assetId
• Compare evaluation requires at least one test case.
• Compare evaluation requires at least two snapshots.

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/18b319810bb50c29. Report an issue: GitHub.