linshenkx/prompt-optimizer · error · EvaluationValidationError

Image compare evaluation requires every compared snapshot to

Error message

Image compare evaluation requires every compared snapshot to include output image evidence.

What it means

Thrown for compare evaluations in image text-2-image mode when at least two snapshots have output image evidence but not every snapshot does. Image compare is all-or-nothing: the service requires the count of snapshots with media to equal the total snapshot count so the comparison is fair.

Source

Thrown at packages/core/src/services/evaluation/service.ts:526

        request.snapshots.forEach((snapshot, index) => {
          this.validateSnapshot(snapshot, `Compare snapshot #${index + 1}`);
          if (!testCaseIds.has(snapshot.testCaseId)) {
            throw new EvaluationValidationError(
              `Compare snapshot #${index + 1} references unknown testCaseId "${snapshot.testCaseId}".`
            );
          }
        });
        if (this.isImageText2ImageMode(request)) {
          const snapshotsWithMedia = request.snapshots.filter((snapshot) =>
            this.hasSnapshotOutputMedia(snapshot)
          );
          if (snapshotsWithMedia.length < 2) {
            throw new EvaluationValidationError(
              'Image compare evaluation requires at least two snapshots with output image evidence.'
            );
          }
          if (snapshotsWithMedia.length !== request.snapshots.length) {
            throw new EvaluationValidationError(
              'Image compare evaluation requires every compared snapshot to include output image evidence.'
            );
          }
        }
        break;

      case 'prompt-only':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        break;

      case 'prompt-iterate':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        if (!request.iterateRequirement?.trim()) {
          throw new EvaluationValidationError('Iteration requirement must not be empty.');

View on GitHub (pinned to 3e677b1d9f)

Solutions

1. Either attach output images to every snapshot or drop the image-less snapshots entirely before comparing
2. Verify every variant run completed image generation before starting the compare
3. Check storage/serialization that one snapshot's media evidence was not lost

Example fix

// before
const snapshots = [imgSnapA, imgSnapB, textSnapC];
await svc.evaluate({ type: 'compare', mode, snapshots, testCases, ... });

// after
const snapshots = [imgSnapA, imgSnapB]; // all have output media
await svc.evaluate({ type: 'compare', mode, snapshots, testCases, ... });

const withMedia = snapshots.filter(hasSnapshotOutputMedia);
if (isImageMode(mode) && withMedia.length !== snapshots.length) {
  // drop image-less snapshots or block the compare
  if (withMedia.length >= 2) snapshots = withMedia; else throw new Error('Incomplete image evidence');
}
await svc.evaluate({ ...req, snapshots });

const everySnapshotHasMedia = (snaps: EvaluationSnapshot[]) => snaps.every(hasSnapshotOutputMedia);

When it happens

Trigger: evaluate({type:'compare', mode:<image text-2-image>}) where some snapshots have output image media and others do not.

Common situations: One variant's image generation failed or is still in progress while others succeeded; heterogeneous compare payload mixing image and text snapshots; evidence field dropped for one snapshot during persistence.

Related errors

• Image compare evaluation requires at least two snapshots wit
• Image result evaluation requires at least one output image e
• Evaluation image evidence "${label}" is missing both assetId
• Compare evaluation requires at least one test case.
• Compare evaluation requires at least two snapshots.

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/e278547fe9afffc4. Report an issue: GitHub.