linshenkx/prompt-optimizer · error · EvaluationValidationError

Result evaluation snapshot testCaseId must match testCase.id

Error message

Result evaluation snapshot testCaseId must match testCase.id.

What it means

Thrown by EvaluationService.validateRequest when a 'result'-type evaluation request has a snapshot whose testCaseId does not equal the provided testCase.id. The service requires that a result evaluation evaluates exactly one snapshot belonging to exactly one test case, so the two identifiers must agree. It surfaces as EvaluationValidationError (code VALIDATION_ERROR) from evaluate/evaluateStream before any LLM call is made.

Source

Thrown at packages/core/src/services/evaluation/service.ts:475

    if (!request.mode) {
      throw new EvaluationValidationError('Evaluation mode configuration must not be empty.');
    }
    if (!request.mode.functionMode) {
      throw new EvaluationValidationError('Function mode must not be empty.');
    }
    if (!request.mode.subMode) {
      throw new EvaluationValidationError('Sub mode must not be empty.');
    }

    switch (request.type) {
      case 'result':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        this.validateTestCase(request.testCase, 'Result evaluation test case');
        this.validateSnapshot(request.snapshot, 'Result evaluation snapshot');
        if (request.snapshot.testCaseId !== request.testCase.id) {
          throw new EvaluationValidationError(
            'Result evaluation snapshot testCaseId must match testCase.id.'
          );
        }
        if (this.isImageText2ImageMode(request) && !this.hasSnapshotOutputMedia(request.snapshot)) {
          throw new EvaluationValidationError(
            'Image result evaluation requires at least one output image evidence item.'
          );
        }
        break;

      case 'compare':
        if (!request.target?.workspacePrompt?.trim()) {
          throw new EvaluationValidationError('Workspace prompt must not be empty.');
        }
        if (!Array.isArray(request.testCases) || request.testCases.length < 1) {
          throw new EvaluationValidationError('Compare evaluation requires at least one test case.');
        }
        if (!Array.isArray(request.snapshots) || request.snapshots.length < 2) {

View on GitHub (pinned to 3e677b1d9f)

Solutions

1. Set snapshot.testCaseId = testCase.id before calling evaluate (they must reference the same test case)
2. Regenerate the snapshot for the current test case instead of reusing a stale one
3. Add a pre-flight assertion in your caller so mismatched ids fail fast with a clearer message

Example fix

// before
const res = await svc.evaluate({
  type: 'result',
  testCase,
  snapshot, // snapshot.testCaseId !== testCase.id
});

// after
const res = await svc.evaluate({
  type: 'result',
  testCase,
  snapshot: { ...snapshot, testCaseId: testCase.id },
});

if (snapshot.testCaseId !== testCase.id) {
  throw new Error(`Snapshot ${snapshot.testCaseId} does not belong to test case ${testCase.id}`);
}
await svc.evaluate({ type: 'result', testCase, snapshot, ... });

const isSnapshotOfCase = (s: EvaluationSnapshot, tc: EvaluationTestCase) =>
  s.testCaseId === tc.id;

try { await svc.evaluate(req); } catch (e) { if (e instanceof EvaluationValidationError && e.message.includes('testCaseId')) syncIdsAndRetry(); else throw e; }

When it happens

Trigger: Calling evaluate({type:'result', ...}) where request.snapshot.testCaseId !== request.testCase.id — e.g. the snapshot was captured under a different test case, or the id was regenerated/re-keyed between building testCase and snapshot.

Common situations: Reusing a snapshot object from a previous run while creating a fresh test case with a new id; copying fixtures and forgetting to sync the ids; frontend passing selected test case separately from the snapshot record it was generated from.

Related errors

• Compare evaluation requires at least one test case.
• Compare evaluation requires at least two snapshots.
• Compare snapshot #${index + 1} references unknown testCaseId
• ${label} id must not be empty.
• Image result evaluation requires at least one output image e

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/59a45ab0a19346b0. Report an issue: GitHub.