mastra-ai/mastra · error · Error

Analysis step failed to produce results

Error message

Analysis step failed to produce results

What it means

The noise-sensitivity scorer's generateScore callback expects results.analyzeStepResult — the structured output of the LLM analyze step. When the analyze step produced nothing (LLM call failed, output didn't parse to analyzeOutputSchema, or the step was skipped), scoring cannot proceed and this error is thrown.

Source

Thrown at packages/evals/src/scorers/llm/noise-sensitivity/index.ts:106

        if (!originalQuery || !noisyResponse) {
          throw new Error('Both original query and noisy response are required for evaluation');
        }

        return createAnalyzePrompt({
          userQuery: originalQuery,
          baselineResponse: options.baselineResponse,
          noisyQuery: options.noisyQuery,
          noisyResponse,
          noiseType: options.noiseType,
        });
      },
    })
    .generateScore(({ results }) => {
      const analysisResult = results.analyzeStepResult;

      if (!analysisResult) {
        throw new Error('Analysis step failed to produce results');
      }

      // Use the LLM's robustness score as primary score
      let finalScore = analysisResult.robustnessScore;

      // Validate score bounds
      finalScore = Math.max(0, Math.min(1, finalScore));

      /**
       * Noise Sensitivity Scoring Algorithm
       *
       * Formula: max(0, min(llm_score, calculated_score) - issues_penalty)
       *
       * Where:
       * - llm_score = direct robustness score from LLM analysis
       * - calculated_score = sum(impact_weights) / num_dimensions
       * - issues_penalty = min(major_issues_count × penalty_rate, max_penalty)
       *

View on GitHub (pinned to 75dd419e61)

Solutions

1. Use a capable judge model that reliably follows the analyzeOutputSchema (e.g. a strong instruct model)
2. Check the model call succeeded (API key, quota, network) — upstream failures surface here as a missing step result
3. Wrap scorer.run in try/catch and retry on transient model failures
4. Log the raw analyze-step output on failure to see why the schema wasn't satisfied

Example fix

// before
const result = await scorer.run({ input, output });
// after
try {
  const result = await scorer.run({ input, output });
} catch (e) {
  if (e.message.includes('Analysis step failed')) {
    // inspect judge model health / raw analyze output, then retry
  }
  throw e;
}

// Pre-check: confirm judge model config is valid and reachable
const healthy = await testJudgeModel(model); // returns false on auth/quota/network failure
if (!healthy) throw new Error('Judge model unavailable; skipping noise-sensitivity scoring');

function hasAnalysisResult(results) {
  return typeof results?.analyzeStepResult === 'object' && results.analyzeStepResult !== null &&
    Number.isFinite(results.analyzeStepResult.robustnessScore);
}

try {
  const result = await scorer.run({ input, output });
} catch (e) {
  if (e.message.includes('Analysis step failed to produce results')) {
    // retry once; judge LLM output is often transiently malformed
    return scorer.run({ input, output });
  }
  throw e;
}

When it happens

Trigger: Running the noise-sensitivity scorer when the analyze LLM step returns no/invalid output — e.g. model returns unstructured text instead of the expected { robustnessScore, dimensions } shape, the model call fails, or the judge model is unavailable.

Common situations: Judge model returning a refusal or hitting a rate limit; an underpowered model that cannot reliably emit the required JSON schema; transient network failures during the analyze step; schema mismatches after upgrading the scorer package.

Related errors

• Analysis step failed to produce results for reason generatio
• MASTR_SCORER_FAILED_TO_RUN_WORKFLOW_FAILED
• Structured output returned no object
• No context available for evaluation
• Both original query and noisy response are required for eval

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/6ecde0c3d3c6935a. Report an issue: GitHub.