{"record":{"id":"c43cd1dcbbdeded0","repo":"mastra-ai/mastra","slug":"both-original-query-and-noisy-response-are-require","errorCode":null,"errorMessage":"Both original query and noisy response are required for evaluation","messagePattern":"Both original query and noisy response are required for evaluation","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/evals/src/scorers/llm/noise-sensitivity/index.ts","lineNumber":90,"sourceCode":"  return createScorer<ScorerRunInputForLLMJudge, ScorerRunOutputForLLMJudge>({\n    id: 'noise-sensitivity-scorer',\n    name: 'Noise Sensitivity (LLM)',\n    description: 'Evaluates how robust an agent is when exposed to irrelevant, distracting, or misleading information',\n    judge: {\n      model,\n      instructions: NOISE_SENSITIVITY_INSTRUCTIONS,\n    },\n    type: 'agent',\n  })\n    .analyze({\n      description: 'Analyze the impact of noise on agent response quality',\n      outputSchema: analyzeOutputSchema,\n      createPrompt: ({ run }) => {\n        const originalQuery = getUserMessageFromRunInput(run.input) ?? '';\n        const noisyResponse = getAssistantMessageFromRunOutput(run.output) ?? '';\n\n        if (!originalQuery || !noisyResponse) {\n          throw new Error('Both original query and noisy response are required for evaluation');\n        }\n\n        return createAnalyzePrompt({\n          userQuery: originalQuery,\n          baselineResponse: options.baselineResponse,\n          noisyQuery: options.noisyQuery,\n          noisyResponse,\n          noiseType: options.noiseType,\n        });\n      },\n    })\n    .generateScore(({ results }) => {\n      const analysisResult = results.analyzeStepResult;\n\n      if (!analysisResult) {\n        throw new Error('Analysis step failed to produce results');\n      }\n","sourceCodeStart":72,"sourceCodeEnd":108,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/evals/src/scorers/llm/noise-sensitivity/index.ts#L72-L108","documentation":"Inside the noise-sensitivity scorer's analyze step, the original query is extracted from run.input and the noisy response from run.output. If either extraction yields an empty value, there is nothing to evaluate robustness against, so the step throws at run time (not construction time).","triggerScenarios":"Running the noise-sensitivity scorer against a run whose input has no user message (getUserMessageFromRunInput returns undefined → '') or whose output has no assistant message (getAssistantMessageFromRunOutput returns undefined → '') — e.g. scoring an aborted/empty generation or a run whose input is only a system prompt.","commonSituations":"Piping workflow/structured run inputs where the user message is nested differently than the extractor expects; evaluating empty assistant responses after a model refusal or API error; running the scorer on the wrong run output field.","solutions":["Ensure the scored run contains a real user message in input and a non-empty assistant message in output","Verify the message-extraction helpers match your run input/output shape (roles, nesting)","Skip runs with empty input/output before invoking the scorer instead of letting it throw mid-evaluation"],"exampleFix":"// before\nawait scorer.run({ input: { messages: systemOnlyMessages }, output });\n// after\nif (!getUserMessageFromRunInput(runInput) || !getAssistantMessageFromRunOutput(runOutput)) {\n  throw new Error('Run must contain a user message and an assistant response');\n}\nawait scorer.run({ input: runInput, output });","handlingStrategy":"validation","validationCode":"const userMsg = getUserMessageFromRunInput(run.input);\nconst asstMsg = getAssistantMessageFromRunOutput(run.output);\nif (!userMsg || !asstMsg) {\n  throw new Error(`Cannot score run: user=${Boolean(userMsg)} assistant=${Boolean(asstMsg)}`);\n}","typeGuard":"function isScoreableRun(io) {\n  return Boolean(getUserMessageFromRunInput(io.input)) && Boolean(getAssistantMessageFromRunOutput(io.output));\n}","tryCatchPattern":"try {\n  const result = await scorer.run({ input, output });\n} catch (e) {\n  if (e.message.includes('original query and noisy response')) {\n    // mark run as unscoreable rather than failing the whole eval batch\n    return { skipped: true, reason: 'empty input/output message' };\n  }\n  throw e;\n}","preventionTips":["Filter runs for empty inputs/outputs before batch scoring","Keep message shapes standard ({ role: 'user'|'assistant'|'system', content }) so extractors work","Capture full conversation runs, not tool-only fragments"],"tags":["evals","runtime","empty-input","message-extraction"],"backgroundTag":"empty-input-validation","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}