mastra-ai/mastra · error · Error

User prompt is required for user prompt alignment scoring

Error message

User prompt is required for user prompt alignment scoring

What it means

The prompt-alignment scorer supports evaluation modes 'user', 'system', and 'both'. In 'user' mode the judge grades the assistant response against the user's prompt; if the run input contains no user message, the scorer throws at run time because there is no user prompt to measure alignment against.

Source

Thrown at packages/evals/src/scorers/llm/prompt-alignment/index.ts:120

    id: 'prompt-alignment-scorer',
    name: 'Prompt Alignment (LLM)',
    description: 'Evaluates how well the agent response aligns with the intent and requirements of the user prompt',
    judge: {
      model,
      instructions: PROMPT_ALIGNMENT_INSTRUCTIONS,
    },
  })
    .analyze({
      description: 'Analyze prompt-response alignment across multiple dimensions',
      outputSchema: analyzeOutputSchema,
      createPrompt: ({ run }) => {
        const userPrompt = getUserMessageFromRunInput(run.input) ?? '';
        const systemPrompt = getCombinedSystemPrompt(run.input) ?? '';
        const agentResponse = getAssistantMessageFromRunOutput(run.output) ?? '';

        // Validation based on evaluation mode
        if (evaluationMode === 'user' && !userPrompt) {
          throw new Error('User prompt is required for user prompt alignment scoring');
        }
        if (evaluationMode === 'system' && !systemPrompt) {
          throw new Error('System prompt is required for system prompt alignment scoring');
        }
        if (evaluationMode === 'both' && !userPrompt && !systemPrompt) {
          throw new Error('A user or system prompt is required for combined alignment scoring');
        }
        if (!agentResponse) {
          throw new Error('Agent response is required for prompt alignment scoring');
        }

        return createAnalyzePrompt({
          userPrompt,
          systemPrompt,
          agentResponse,
          evaluationMode,
          conversationHistory: historyOptions && getConversationHistoryFromRunInput(run.input, historyOptions),
        });

View on GitHub (pinned to 75dd419e61)

Solutions

1. Ensure run.input includes at least one user-role message
2. If your prompts live in the system message, set evaluationMode to 'system' or 'both'
3. Normalize roles before scoring (map 'human' → 'user') if using non-standard message shapes

Example fix

// before
await scorer.run({ input: { messages: [{ role: 'system', content: 'Do X' }] }, output });
// after
await scorer.run({ input: { messages: [{ role: 'system', content: 'Do X' }, { role: 'user', content: 'the actual request' }] }, output });

if (evaluationMode === 'user' && !getUserMessageFromRunInput(run.input)) {
  throw new Error('Run has no user prompt; cannot score user-prompt alignment');
}

function hasUserPrompt(input) {
  return Boolean(getUserMessageFromRunInput(input));
}

try {
  const result = await scorer.run({ input, output });
} catch (e) {
  if (e.message.includes('User prompt is required')) {
    return { skipped: true, reason: 'run has no user prompt' };
  }
  throw e;
}

When it happens

Trigger: Running createPromptAlignmentScorerLLM with options.evaluationMode = 'user' (the default path) on a run whose input has no user-role message — getUserMessageFromRunInput returns undefined/empty.

Common situations: Scoring runs whose input is only a system prompt (e.g. classification setups that put everything in system); message arrays using non-standard roles ('human' instead of 'user') that the extractor doesn't recognize; passing pre-formatted prompt strings instead of message arrays.

Related errors

• System prompt is required for system prompt alignment scorin
• A user or system prompt is required for combined alignment s
• Both original query and noisy response are required for eval
• RUN_EXPERIMENT_FAILED_NO_DATA_PROVIDED
• INVALID_DATA_ITEM

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/dda31abc4965cd72. Report an issue: GitHub.