{"record":{"id":"ddf7e66b13235276","repo":"mastra-ai/mastra","slug":"both-baselineresponse-and-noisyquery-are-required","errorCode":null,"errorMessage":"Both baselineResponse and noisyQuery are required for Noise Sensitivity scoring","messagePattern":"Both baselineResponse and noisyQuery are required for Noise Sensitivity scoring","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/evals/src/scorers/llm/noise-sensitivity/index.ts","lineNumber":69,"sourceCode":"  significant: 0.3,\n  severe: 0.1,\n} as const;\n\nconst DEFAULT_SCORING = {\n  MAJOR_ISSUE_PENALTY_PER_ITEM: 0.1, // 10% penalty per major issue\n  MAX_MAJOR_ISSUE_PENALTY: 0.3, // Maximum 30% penalty for major issues\n  DISCREPANCY_THRESHOLD: 0.2, // Threshold for choosing conservative score\n} as const;\n\nexport function createNoiseSensitivityScorerLLM({\n  model,\n  options,\n}: {\n  model: MastraModelConfig;\n  options: NoiseSensitivityOptions;\n}) {\n  if (!options.baselineResponse || !options.noisyQuery) {\n    throw new Error('Both baselineResponse and noisyQuery are required for Noise Sensitivity scoring');\n  }\n\n  return createScorer<ScorerRunInputForLLMJudge, ScorerRunOutputForLLMJudge>({\n    id: 'noise-sensitivity-scorer',\n    name: 'Noise Sensitivity (LLM)',\n    description: 'Evaluates how robust an agent is when exposed to irrelevant, distracting, or misleading information',\n    judge: {\n      model,\n      instructions: NOISE_SENSITIVITY_INSTRUCTIONS,\n    },\n    type: 'agent',\n  })\n    .analyze({\n      description: 'Analyze the impact of noise on agent response quality',\n      outputSchema: analyzeOutputSchema,\n      createPrompt: ({ run }) => {\n        const originalQuery = getUserMessageFromRunInput(run.input) ?? '';\n        const noisyResponse = getAssistantMessageFromRunOutput(run.output) ?? '';","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/evals/src/scorers/llm/noise-sensitivity/index.ts#L51-L87","documentation":"createNoiseSensitivityScorerLLM compares the agent's response under noisy conditions against two required fixtures: a `baselineResponse` (the answer without noise) and a `noisyQuery` (the query containing distracting/misleading content). If either is missing the comparison is impossible, so the factory throws synchronously.","triggerScenarios":"Calling createNoiseSensitivityScorerLLM({ model, options }) where options.baselineResponse is falsy (undefined/null/empty string) OR options.noisyQuery is falsy — e.g. { model, options: { baselineResponse: '...' } } without noisyQuery.","commonSituations":"Only partially migrating an older options shape that used different field names; a generator script that produced the baseline failing and returning undefined; forgetting that both fixtures are required, not optional tuning knobs.","solutions":["Provide both options.baselineResponse and options.noisyQuery as non-empty strings","Generate the baselineResponse by running the agent on the clean query first, then pass it in","Add a pre-construction check: if (!baselineResponse || !noisyQuery) throw before calling the factory"],"exampleFix":"// before\nconst scorer = createNoiseSensitivityScorerLLM({ model, options: { noisyQuery } });\n// after\nconst scorer = createNoiseSensitivityScorerLLM({\n  model,\n  options: { baselineResponse: await runAgent(cleanQuery), noisyQuery },\n});","handlingStrategy":"validation","validationCode":"if (!opts.baselineResponse || !opts.noisyQuery) {\n  throw new Error('Noise sensitivity scoring needs both baselineResponse and noisyQuery');\n}","typeGuard":"function hasNoiseFixtures(o) {\n  return typeof o?.baselineResponse === 'string' && o.baselineResponse.length > 0 &&\n         typeof o?.noisyQuery === 'string' && o.noisyQuery.length > 0;\n}","tryCatchPattern":"try {\n  const scorer = createNoiseSensitivityScorerLLM({ model, options });\n} catch (e) {\n  if (e.message.includes('baselineResponse and noisyQuery')) {\n    throw new Error('Missing noise-sensitivity fixtures; generate the baseline first', { cause: e });\n  }\n  throw e;\n}","preventionTips":["Generate baselineResponse programmatically (run the agent on the clean query) so it is never undefined","Keep noisyQuery fixtures in versioned test data, not ad-hoc strings","Type options with required fields so TypeScript catches omissions"],"tags":["configuration","validation","evals","missing-parameter"],"backgroundTag":"missing-required-scorer-option","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T08:17:16.595Z"}