mastra-ai/mastra · error · Error

Both baselineResponse and noisyQuery are required for Noise

Error message

Both baselineResponse and noisyQuery are required for Noise Sensitivity scoring

What it means

createNoiseSensitivityScorerLLM compares the agent's response under noisy conditions against two required fixtures: a `baselineResponse` (the answer without noise) and a `noisyQuery` (the query containing distracting/misleading content). If either is missing the comparison is impossible, so the factory throws synchronously.

Source

Thrown at packages/evals/src/scorers/llm/noise-sensitivity/index.ts:69

  significant: 0.3,
  severe: 0.1,
} as const;

const DEFAULT_SCORING = {
  MAJOR_ISSUE_PENALTY_PER_ITEM: 0.1, // 10% penalty per major issue
  MAX_MAJOR_ISSUE_PENALTY: 0.3, // Maximum 30% penalty for major issues
  DISCREPANCY_THRESHOLD: 0.2, // Threshold for choosing conservative score
} as const;

export function createNoiseSensitivityScorerLLM({
  model,
  options,
}: {
  model: MastraModelConfig;
  options: NoiseSensitivityOptions;
}) {
  if (!options.baselineResponse || !options.noisyQuery) {
    throw new Error('Both baselineResponse and noisyQuery are required for Noise Sensitivity scoring');
  }

  return createScorer<ScorerRunInputForLLMJudge, ScorerRunOutputForLLMJudge>({
    id: 'noise-sensitivity-scorer',
    name: 'Noise Sensitivity (LLM)',
    description: 'Evaluates how robust an agent is when exposed to irrelevant, distracting, or misleading information',
    judge: {
      model,
      instructions: NOISE_SENSITIVITY_INSTRUCTIONS,
    },
    type: 'agent',
  })
    .analyze({
      description: 'Analyze the impact of noise on agent response quality',
      outputSchema: analyzeOutputSchema,
      createPrompt: ({ run }) => {
        const originalQuery = getUserMessageFromRunInput(run.input) ?? '';
        const noisyResponse = getAssistantMessageFromRunOutput(run.output) ?? '';

View on GitHub (pinned to 75dd419e61)

Solutions

1. Provide both options.baselineResponse and options.noisyQuery as non-empty strings
2. Generate the baselineResponse by running the agent on the clean query first, then pass it in
3. Add a pre-construction check: if (!baselineResponse || !noisyQuery) throw before calling the factory

Example fix

// before
const scorer = createNoiseSensitivityScorerLLM({ model, options: { noisyQuery } });
// after
const scorer = createNoiseSensitivityScorerLLM({
  model,
  options: { baselineResponse: await runAgent(cleanQuery), noisyQuery },
});

if (!opts.baselineResponse || !opts.noisyQuery) {
  throw new Error('Noise sensitivity scoring needs both baselineResponse and noisyQuery');
}

function hasNoiseFixtures(o) {
  return typeof o?.baselineResponse === 'string' && o.baselineResponse.length > 0 &&
         typeof o?.noisyQuery === 'string' && o.noisyQuery.length > 0;
}

try {
  const scorer = createNoiseSensitivityScorerLLM({ model, options });
} catch (e) {
  if (e.message.includes('baselineResponse and noisyQuery')) {
    throw new Error('Missing noise-sensitivity fixtures; generate the baseline first', { cause: e });
  }
  throw e;
}

When it happens

Trigger: Calling createNoiseSensitivityScorerLLM({ model, options }) where options.baselineResponse is falsy (undefined/null/empty string) OR options.noisyQuery is falsy — e.g. { model, options: { baselineResponse: '...' } } without noisyQuery.

Common situations: Only partially migrating an older options shape that used different field names; a generator script that produced the baseline failing and returning undefined; forgetting that both fixtures are required, not optional tuning knobs.

Related errors

• Either context or contextExtractor is required for Context R
• NO_SCORERS_PROVIDED
• Context array cannot be empty if provided
• createMultiTurnJudgeScorer: options.scale must be a finite n
• Google RBAC roleMapping is required.

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/ddf7e66b13235276. Report an issue: GitHub.