mastra-ai/mastra · error · Error

Both original query and noisy response are required for eval

Error message

Both original query and noisy response are required for evaluation

What it means

Inside the noise-sensitivity scorer's analyze step, the original query is extracted from run.input and the noisy response from run.output. If either extraction yields an empty value, there is nothing to evaluate robustness against, so the step throws at run time (not construction time).

Source

Thrown at packages/evals/src/scorers/llm/noise-sensitivity/index.ts:90

  return createScorer<ScorerRunInputForLLMJudge, ScorerRunOutputForLLMJudge>({
    id: 'noise-sensitivity-scorer',
    name: 'Noise Sensitivity (LLM)',
    description: 'Evaluates how robust an agent is when exposed to irrelevant, distracting, or misleading information',
    judge: {
      model,
      instructions: NOISE_SENSITIVITY_INSTRUCTIONS,
    },
    type: 'agent',
  })
    .analyze({
      description: 'Analyze the impact of noise on agent response quality',
      outputSchema: analyzeOutputSchema,
      createPrompt: ({ run }) => {
        const originalQuery = getUserMessageFromRunInput(run.input) ?? '';
        const noisyResponse = getAssistantMessageFromRunOutput(run.output) ?? '';

        if (!originalQuery || !noisyResponse) {
          throw new Error('Both original query and noisy response are required for evaluation');
        }

        return createAnalyzePrompt({
          userQuery: originalQuery,
          baselineResponse: options.baselineResponse,
          noisyQuery: options.noisyQuery,
          noisyResponse,
          noiseType: options.noiseType,
        });
      },
    })
    .generateScore(({ results }) => {
      const analysisResult = results.analyzeStepResult;

      if (!analysisResult) {
        throw new Error('Analysis step failed to produce results');
      }

View on GitHub (pinned to 75dd419e61)

Solutions

1. Ensure the scored run contains a real user message in input and a non-empty assistant message in output
2. Verify the message-extraction helpers match your run input/output shape (roles, nesting)
3. Skip runs with empty input/output before invoking the scorer instead of letting it throw mid-evaluation

Example fix

// before
await scorer.run({ input: { messages: systemOnlyMessages }, output });
// after
if (!getUserMessageFromRunInput(runInput) || !getAssistantMessageFromRunOutput(runOutput)) {
  throw new Error('Run must contain a user message and an assistant response');
}
await scorer.run({ input: runInput, output });

const userMsg = getUserMessageFromRunInput(run.input);
const asstMsg = getAssistantMessageFromRunOutput(run.output);
if (!userMsg || !asstMsg) {
  throw new Error(`Cannot score run: user=${Boolean(userMsg)} assistant=${Boolean(asstMsg)}`);
}

function isScoreableRun(io) {
  return Boolean(getUserMessageFromRunInput(io.input)) && Boolean(getAssistantMessageFromRunOutput(io.output));
}

try {
  const result = await scorer.run({ input, output });
} catch (e) {
  if (e.message.includes('original query and noisy response')) {
    // mark run as unscoreable rather than failing the whole eval batch
    return { skipped: true, reason: 'empty input/output message' };
  }
  throw e;
}

When it happens

Trigger: Running the noise-sensitivity scorer against a run whose input has no user message (getUserMessageFromRunInput returns undefined → '') or whose output has no assistant message (getAssistantMessageFromRunOutput returns undefined → '') — e.g. scoring an aborted/empty generation or a run whose input is only a system prompt.

Common situations: Piping workflow/structured run inputs where the user message is nested differently than the extractor expects; evaluating empty assistant responses after a model refusal or API error; running the scorer on the wrong run output field.

Related errors

• User prompt is required for user prompt alignment scoring
• System prompt is required for system prompt alignment scorin
• A user or system prompt is required for combined alignment s
• RUN_EXPERIMENT_FAILED_NO_DATA_PROVIDED
• Input and output messages cannot be null or empty

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/c43cd1dcbbdeded0. Report an issue: GitHub.