{"record":{"id":"00689ab2af0af0da","repo":"mastra-ai/mastra","slug":"answer-similarity-scorer-requires-ground-truth-to","errorCode":null,"errorMessage":"Answer Similarity Scorer requires ground truth to be provided","messagePattern":"Answer Similarity Scorer requires ground truth to be provided","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/evals/src/scorers/llm/answer-similarity/index.ts","lineNumber":94,"sourceCode":"  const mergedOptions = { ...ANSWER_SIMILARITY_DEFAULT_OPTIONS, ...options };\n  return createScorer<ScorerRunInputForLLMJudge, ScorerRunOutputForLLMJudge>({\n    id: 'answer-similarity-scorer',\n    name: 'Answer Similarity Scorer',\n    description: 'Evaluates how similar an agent output is to a ground truth answer for CI/CD testing',\n    judge: {\n      model,\n      instructions: ANSWER_SIMILARITY_INSTRUCTIONS,\n    },\n    type: 'agent',\n  })\n    .preprocess({\n      description: 'Extract semantic units from output and ground truth',\n      outputSchema: extractOutputSchema,\n      createPrompt: ({ run }) => {\n        // Check if ground truth exists\n        if (!run.groundTruth) {\n          if (mergedOptions.requireGroundTruth) {\n            throw new Error('Answer Similarity Scorer requires ground truth to be provided');\n          }\n          // If ground truth is not required and missing, return empty units\n          return createExtractPrompt({\n            output: '',\n            groundTruth: '',\n          });\n        }\n\n        const output = getAssistantMessageFromRunOutput(run.output) ?? '';\n        const groundTruth = typeof run.groundTruth === 'string' ? run.groundTruth : JSON.stringify(run.groundTruth);\n\n        return createExtractPrompt({\n          output,\n          groundTruth,\n        });\n      },\n    })\n    .analyze({","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/evals/src/scorers/llm/answer-similarity/index.ts#L76-L112","documentation":"The answer similarity scorer compares the output against a ground truth using an LLM-judge extraction step. When the run has no groundTruth and the merged options set requireGroundTruth: true, the extraction step throws instead of proceeding with empty units, because similarity against nothing is meaningless. When requireGroundTruth is false (default), it degrades gracefully by extracting from empty strings.","triggerScenarios":"Calling the scorer (scorer/scorerStrict/customScorer built with requireGroundTruth: true) via run() or score() on a run object that omits run.groundTruth; creating test fixtures without groundTruth while using the strict variants.","commonSituations":"Using scorerStrict or a custom scorer with requireGroundTruth enabled but forgetting to supply groundTruth in the scoring input; batch evaluation scripts where some rows lack the expected-answer column; migrating from the lenient scorer to the strict one without updating payloads.","solutions":["Provide groundTruth on the run/input object when calling the scorer, e.g. { input, output, groundTruth: 'expected answer' }","Use the default (lenient) scorer if ground truth is optional in your dataset","Filter out dataset rows without ground truth before scoring when requireGroundTruth is enabled","Verify the groundTruth key spelling and that you are passing it in the run payload, not a separate argument"],"exampleFix":"// before\nawait scorerStrict.run({ input: { inputMessages }, output });\n\n// after\nawait scorerStrict.run({ input: { inputMessages }, output, groundTruth: 'Paris is the capital of France' });","handlingStrategy":"validation","validationCode":"if (usingStrictVariant && !run.groundTruth) {\n  throw new Error('Strict answer-similarity scorer requires groundTruth');\n}\nawait scorerStrict.run({ input: { inputMessages }, output, groundTruth });","typeGuard":"function hasGroundTruth(r: { groundTruth?: string }): r is { groundTruth: string } {\n  return typeof r.groundTruth === 'string' && r.groundTruth.length > 0;\n}","tryCatchPattern":"try {\n  await scorerStrict.run(payload);\n} catch (err) {\n  if ((err as Error).message.includes('requires ground truth')) {\n    console.warn('Row skipped: missing groundTruth');\n    return null;\n  }\n  throw err;\n}","preventionTips":["Include groundTruth in every scoring payload when using strict variants","Choose the lenient default scorer for datasets with partial ground truth","Validate dataset rows (groundTruth column present) before batch scoring","Pass groundTruth in the run payload, not as a separate argument"],"tags":["evals","scorer","llm-judge","answer-similarity","ground-truth"],"backgroundTag":"missing-ground-truth","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}