mastra-ai/mastra · error · Error

System prompt is required for system prompt alignment scorin

Error message

System prompt is required for system prompt alignment scoring

What it means

In 'system' evaluation mode the prompt-alignment scorer grades the response against the combined system prompt. If getCombinedSystemPrompt(run.input) returns empty, there is no system prompt to judge against, so the scorer throws at run time.

Source

Thrown at packages/evals/src/scorers/llm/prompt-alignment/index.ts:123

    judge: {
      model,
      instructions: PROMPT_ALIGNMENT_INSTRUCTIONS,
    },
  })
    .analyze({
      description: 'Analyze prompt-response alignment across multiple dimensions',
      outputSchema: analyzeOutputSchema,
      createPrompt: ({ run }) => {
        const userPrompt = getUserMessageFromRunInput(run.input) ?? '';
        const systemPrompt = getCombinedSystemPrompt(run.input) ?? '';
        const agentResponse = getAssistantMessageFromRunOutput(run.output) ?? '';

        // Validation based on evaluation mode
        if (evaluationMode === 'user' && !userPrompt) {
          throw new Error('User prompt is required for user prompt alignment scoring');
        }
        if (evaluationMode === 'system' && !systemPrompt) {
          throw new Error('System prompt is required for system prompt alignment scoring');
        }
        if (evaluationMode === 'both' && !userPrompt && !systemPrompt) {
          throw new Error('A user or system prompt is required for combined alignment scoring');
        }
        if (!agentResponse) {
          throw new Error('Agent response is required for prompt alignment scoring');
        }

        return createAnalyzePrompt({
          userPrompt,
          systemPrompt,
          agentResponse,
          evaluationMode,
          conversationHistory: historyOptions && getConversationHistoryFromRunInput(run.input, historyOptions),
        });
      },
    })
    .generateScore(({ results }) => {

View on GitHub (pinned to 75dd419e61)

Solutions

1. Ensure run.input contains a system-role message when using evaluationMode 'system'
2. Switch evaluationMode to 'user' if your runs have no system prompt
3. Verify getCombinedSystemPrompt's expected message format matches your run input structure

Example fix

// before
const scorer = createPromptAlignmentScorerLLM({ model, options: { evaluationMode: 'system' } });
await scorer.run({ input: { messages: [{ role: 'user', content: 'hi' }] }, output });
// after
const mode = hasSystemPrompt(runInput) ? 'system' : 'user';
const scorer = createPromptAlignmentScorerLLM({ model, options: { evaluationMode: mode } });

if (evaluationMode === 'system' && !getCombinedSystemPrompt(run.input)) {
  throw new Error('Run has no system prompt; cannot score system-prompt alignment');
}

function hasSystemPrompt(input) {
  return Boolean(getCombinedSystemPrompt(input));
}

try {
  const result = await scorer.run({ input, output });
} catch (e) {
  if (e.message.includes('System prompt is required')) {
    return { skipped: true, reason: 'run has no system prompt' };
  }
  throw e;
}

When it happens

Trigger: Running the scorer with evaluationMode = 'system' on a run whose input contains no system-role messages — e.g. a plain user/assistant exchange scored with the wrong mode.

Common situations: Selecting mode 'system' in config while the scored conversations have no system message; system instructions embedded in the first user message instead of a system role; misconfigured extractor that ignores the system role.

Related errors

• User prompt is required for user prompt alignment scoring
• A user or system prompt is required for combined alignment s
• Both original query and noisy response are required for eval
• RUN_EXPERIMENT_FAILED_NO_DATA_PROVIDED
• INVALID_DATA_ITEM

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/81bc015f7f58e459. Report an issue: GitHub.