{"record":{"id":"6ecde0c3d3c6935a","repo":"mastra-ai/mastra","slug":"analysis-step-failed-to-produce-results","errorCode":null,"errorMessage":"Analysis step failed to produce results","messagePattern":"Analysis step failed to produce results","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/evals/src/scorers/llm/noise-sensitivity/index.ts","lineNumber":106,"sourceCode":"\n        if (!originalQuery || !noisyResponse) {\n          throw new Error('Both original query and noisy response are required for evaluation');\n        }\n\n        return createAnalyzePrompt({\n          userQuery: originalQuery,\n          baselineResponse: options.baselineResponse,\n          noisyQuery: options.noisyQuery,\n          noisyResponse,\n          noiseType: options.noiseType,\n        });\n      },\n    })\n    .generateScore(({ results }) => {\n      const analysisResult = results.analyzeStepResult;\n\n      if (!analysisResult) {\n        throw new Error('Analysis step failed to produce results');\n      }\n\n      // Use the LLM's robustness score as primary score\n      let finalScore = analysisResult.robustnessScore;\n\n      // Validate score bounds\n      finalScore = Math.max(0, Math.min(1, finalScore));\n\n      /**\n       * Noise Sensitivity Scoring Algorithm\n       *\n       * Formula: max(0, min(llm_score, calculated_score) - issues_penalty)\n       *\n       * Where:\n       * - llm_score = direct robustness score from LLM analysis\n       * - calculated_score = sum(impact_weights) / num_dimensions\n       * - issues_penalty = min(major_issues_count × penalty_rate, max_penalty)\n       *","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/evals/src/scorers/llm/noise-sensitivity/index.ts#L88-L124","documentation":"The noise-sensitivity scorer's generateScore callback expects results.analyzeStepResult — the structured output of the LLM analyze step. When the analyze step produced nothing (LLM call failed, output didn't parse to analyzeOutputSchema, or the step was skipped), scoring cannot proceed and this error is thrown.","triggerScenarios":"Running the noise-sensitivity scorer when the analyze LLM step returns no/invalid output — e.g. model returns unstructured text instead of the expected { robustnessScore, dimensions } shape, the model call fails, or the judge model is unavailable.","commonSituations":"Judge model returning a refusal or hitting a rate limit; an underpowered model that cannot reliably emit the required JSON schema; transient network failures during the analyze step; schema mismatches after upgrading the scorer package.","solutions":["Use a capable judge model that reliably follows the analyzeOutputSchema (e.g. a strong instruct model)","Check the model call succeeded (API key, quota, network) — upstream failures surface here as a missing step result","Wrap scorer.run in try/catch and retry on transient model failures","Log the raw analyze-step output on failure to see why the schema wasn't satisfied"],"exampleFix":"// before\nconst result = await scorer.run({ input, output });\n// after\ntry {\n  const result = await scorer.run({ input, output });\n} catch (e) {\n  if (e.message.includes('Analysis step failed')) {\n    // inspect judge model health / raw analyze output, then retry\n  }\n  throw e;\n}","handlingStrategy":"try-catch","validationCode":"// Pre-check: confirm judge model config is valid and reachable\nconst healthy = await testJudgeModel(model); // returns false on auth/quota/network failure\nif (!healthy) throw new Error('Judge model unavailable; skipping noise-sensitivity scoring');","typeGuard":"function hasAnalysisResult(results) {\n  return typeof results?.analyzeStepResult === 'object' && results.analyzeStepResult !== null &&\n    Number.isFinite(results.analyzeStepResult.robustnessScore);\n}","tryCatchPattern":"try {\n  const result = await scorer.run({ input, output });\n} catch (e) {\n  if (e.message.includes('Analysis step failed to produce results')) {\n    // retry once; judge LLM output is often transiently malformed\n    return scorer.run({ input, output });\n  }\n  throw e;\n}","preventionTips":["Use a strong instruct model as the judge and request schema-constrained output","Retry scorer runs on transient LLM failures with backoff","Monitor judge-model refusals and rate limits in eval dashboards","Log raw analyze-step output when scoring fails to debug schema mismatches"],"tags":["evals","llm","runtime","schema-validation"],"backgroundTag":"llm-structured-output-missing","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T08:17:16.595Z"}