mastra-ai/mastra · error · MastraError

RUN_EXPERIMENT_SCORER_FAILED_TO_SCORE_RESULT

RUN_EXPERIMENT_SCORER_FAILED_TO_SCORE_RESULT

Error message

Failed to run experiment: Error running scorer ${scorer.id}

What it means

Thrown when a regular (non-agent, non-trajectory) scorer throws while scoring the target's result during an experiment run. The scorer run (scorer.run with target trace/span ids) raised, and the runner wraps it in a MastraError naming the failing scorer id. Category USER, domain SCORER.

Source

Thrown at packages/core/src/evals/run/index.ts:1101

  if (Array.isArray(scorers)) {
    for (const scorer of scorers) {
      try {
        const score = await scorer.run({
          input: targetResult.scoringData?.input,
          output: targetResult.scoringData?.output,
          groundTruth: item.groundTruth,
          requestContext: item.requestContext,
          scoreSource: 'experiment',
          targetScope: 'span',
          targetEntityType,
          targetTraceId,
          targetSpanId: targetResult.spanId,
        });

        scorerResults[scorer.id] = score;
      } catch (error) {
        throw new MastraError(
          {
            domain: 'SCORER',
            id: 'RUN_EXPERIMENT_SCORER_FAILED_TO_SCORE_RESULT',
            category: 'USER',
            text: `Failed to run experiment: Error running scorer ${scorer.id}`,
            details: {
              scorerId: scorer.id,
              item: JSON.stringify(item),
            },
          },
          error,
        );
      }
    }
  } else if (isAgentScorerConfig(scorers)) {
    // Handle agent scorer config (agent-level + trajectory scorers)
    if (scorers.agent) {
      const agentScorerResults: Record<string, any> = {};

View on GitHub (pinned to 75dd419e61)

Solutions

Read the wrapped cause to see the scorer's internal error.
Run the scorer directly via mastra.getScorer(...).run(...) against a saved trace/span to reproduce.
Check the scorer's judge model config (API key, model id) if it is an LLM-based scorer.
Ensure required scorer inputs (e.g. reference) exist on every dataset item.
Add defensive checks in the scorer's process/extract for missing fields.

Example fix

// before
scorers: { correctness } // judge model key missing at runtime
// after: validate env before the run
if (!process.env.OPENAI_API_KEY) throw new Error('OPENAI_API_KEY required for correctness scorer');
scorers: { correctness }

Defensive patterns

Strategy: try-catch

Validate before calling

// smoke-test each scorer once before a full experiment
await scorer.run({ input: 'ping', output: 'pong' });

Type guard

function isScorerError(e) {
  return e instanceof Error && typeof e.message === 'string' && e.message.includes('Error running scorer');
}

Try / catch

try {
  await experiment.run();
} catch (e) {
  if (e?.id === 'RUN_EXPERIMENT_SCORER_FAILED_TO_SCORE_RESULT') {
    console.error('Scorer failed, cause:', e.cause);
    // optionally continue the run without this scorer
  } else throw e;
}

Prevention

Smoke-test LLM-judge scorers with one sample before large runs
Ensure judge model env keys are set for every scorer
Guard scorer extract/process against missing fields (reference, output)
Run scorers standalone via scorer.run on a saved result to verify

When it happens

Trigger: In runEvals/experiment execution, iterating experiment.scorers and calling scorer.run(...) on the target result: the scorer's process function throws (LLM call failure for LLM-judge scorers, reference data missing, extract/output schema mismatch).

Common situations: LLM-judge scorer using a misconfigured/unavailable model or API key; scorer extract returning undefined so process fails; scorer expecting `reference` but the dataset item lacks it; custom scorer code with a runtime bug.

Related errors

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/4037fdd936241535. Report an issue: GitHub.