mastra-ai/mastra · error · Error

Answer Similarity Scorer requires ground truth to be provide

Error message

Answer Similarity Scorer requires ground truth to be provided

What it means

The answer similarity scorer compares the output against a ground truth using an LLM-judge extraction step. When the run has no groundTruth and the merged options set requireGroundTruth: true, the extraction step throws instead of proceeding with empty units, because similarity against nothing is meaningless. When requireGroundTruth is false (default), it degrades gracefully by extracting from empty strings.

Source

Thrown at packages/evals/src/scorers/llm/answer-similarity/index.ts:94

  const mergedOptions = { ...ANSWER_SIMILARITY_DEFAULT_OPTIONS, ...options };
  return createScorer<ScorerRunInputForLLMJudge, ScorerRunOutputForLLMJudge>({
    id: 'answer-similarity-scorer',
    name: 'Answer Similarity Scorer',
    description: 'Evaluates how similar an agent output is to a ground truth answer for CI/CD testing',
    judge: {
      model,
      instructions: ANSWER_SIMILARITY_INSTRUCTIONS,
    },
    type: 'agent',
  })
    .preprocess({
      description: 'Extract semantic units from output and ground truth',
      outputSchema: extractOutputSchema,
      createPrompt: ({ run }) => {
        // Check if ground truth exists
        if (!run.groundTruth) {
          if (mergedOptions.requireGroundTruth) {
            throw new Error('Answer Similarity Scorer requires ground truth to be provided');
          }
          // If ground truth is not required and missing, return empty units
          return createExtractPrompt({
            output: '',
            groundTruth: '',
          });
        }

        const output = getAssistantMessageFromRunOutput(run.output) ?? '';
        const groundTruth = typeof run.groundTruth === 'string' ? run.groundTruth : JSON.stringify(run.groundTruth);

        return createExtractPrompt({
          output,
          groundTruth,
        });
      },
    })
    .analyze({

View on GitHub (pinned to 75dd419e61)

Solutions

1. Provide groundTruth on the run/input object when calling the scorer, e.g. { input, output, groundTruth: 'expected answer' }
2. Use the default (lenient) scorer if ground truth is optional in your dataset
3. Filter out dataset rows without ground truth before scoring when requireGroundTruth is enabled
4. Verify the groundTruth key spelling and that you are passing it in the run payload, not a separate argument

Example fix

// before
await scorerStrict.run({ input: { inputMessages }, output });

// after
await scorerStrict.run({ input: { inputMessages }, output, groundTruth: 'Paris is the capital of France' });

if (usingStrictVariant && !run.groundTruth) {
  throw new Error('Strict answer-similarity scorer requires groundTruth');
}
await scorerStrict.run({ input: { inputMessages }, output, groundTruth });

function hasGroundTruth(r: { groundTruth?: string }): r is { groundTruth: string } {
  return typeof r.groundTruth === 'string' && r.groundTruth.length > 0;
}

try {
  await scorerStrict.run(payload);
} catch (err) {
  if ((err as Error).message.includes('requires ground truth')) {
    console.warn('Row skipped: missing groundTruth');
    return null;
  }
  throw err;
}

When it happens

Trigger: Calling the scorer (scorer/scorerStrict/customScorer built with requireGroundTruth: true) via run() or score() on a run object that omits run.groundTruth; creating test fixtures without groundTruth while using the strict variants.

Common situations: Using scorerStrict or a custom scorer with requireGroundTruth enabled but forgetting to supply groundTruth in the scoring input; batch evaluation scripts where some rows lack the expected-answer column; migrating from the lenient scorer to the strict one without updating payloads.

Related errors

• MASTR_SCORER_FAILED_TO_CREATE_MISSING_ID
• MASTR_SCORER_FAILED_TO_RUN_MISSING_GENERATE_SCORE
• MASTR_SCORER_FAILED_TO_RUN_WORKFLOW_FAILED
• Step "${scorerStep.name}" is not a prompt object
• RUN_EXPERIMENT_SCORER_FAILED_TO_SCORE_RESULT

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/00689ab2af0af0da. Report an issue: GitHub.