mastra-ai/mastra · error · Error

Input and output messages cannot be null or empty

Error message

Input and output messages cannot be null or empty

What it means

The tool-call-accuracy scorer needs both the conversation input messages (run.input.inputMessages) and the output messages to extract which tools were actually called. If either is null, undefined, or an empty array, the scorer's preprocess step throws because no accuracy comparison is possible. This is a runtime guard inside the scorer pipeline, not the constructor.

Source

Thrown at packages/evals/src/scorers/llm/tool-call-accuracy/index.ts:50

export function createToolCallAccuracyScorerLLM({ model, availableTools }: ToolCallAccuracyOptions) {
  const toolDefinitions = availableTools.map(tool => `${tool.id}: ${tool.description}`).join('\n');

  return createScorer({
    id: 'llm-tool-call-accuracy-scorer',
    name: 'Tool Call Accuracy (LLM)',
    description: 'Evaluates whether an agent selected appropriate tools for the given task using LLM analysis',
    judge: {
      model,
      instructions: TOOL_SELECTION_ACCURACY_INSTRUCTIONS,
    },
    type: 'agent',
  })
    .preprocess(async ({ run }) => {
      const isInputInvalid = !run.input || !run.input.inputMessages || run.input.inputMessages.length === 0;
      const isOutputInvalid = !run.output || run.output.length === 0;

      if (isInputInvalid || isOutputInvalid) {
        throw new Error('Input and output messages cannot be null or empty');
      }

      const { tools: actualTools, toolCallInfos } = extractToolCalls(run.output);

      return {
        actualTools,
        hasToolCalls: actualTools.length > 0,
        toolCallInfos,
      };
    })
    .analyze({
      description: 'Analyze the appropriateness of tool selections',
      outputSchema: analyzeOutputSchema,
      createPrompt: ({ run, results }) => {
        const userInput = getUserMessageFromRunInput(run.input) ?? '';
        const agentResponse = getAssistantMessageFromRunOutput(run.output) ?? '';

        const toolsCalled = results.preprocessStepResult?.actualTools || [];

View on GitHub (pinned to 75dd419e61)

Solutions

1. Verify the run you pass has non-empty inputMessages and output arrays before scoring.
2. If the run genuinely produced no output, skip it — filter runs with output.length === 0 out of your eval set.
3. Check how you build run.input; inputMessages must be under run.input.inputMessages, not a sibling key.

Example fix

// before
for (const run of runs) {
  await scorer.run(run);
}
// after
for (const run of runs) {
  const ok = run?.input?.inputMessages?.length && run?.output?.length;
  if (!ok) continue;
  await scorer.run(run);
}

function isScorableRun(run: { input?: { inputMessages?: unknown[] }; output?: unknown[] }): boolean {
  return Boolean(run?.input?.inputMessages?.length && run?.output?.length);
}

const isNonEmptyArray = <T>(a: T[] | null | undefined): a is T[] => Array.isArray(a) && a.length > 0;

try {
  await scorer.run(run);
} catch (err) {
  if (err instanceof Error && err.message.includes('cannot be null or empty')) {
    logger.warn({ runId: run.id }, 'skipping run with empty input/output');
  } else throw err;
}

When it happens

Trigger: Invoking the tool-call-accuracy scorer with a run whose run.input.inputMessages is missing/empty or whose run.output array is empty — e.g. scoring a run that produced no output, passing a malformed ScorerRunInputForLLMJudge, or fetching a run from storage before the output was persisted.

Common situations: Scoring partially-failed or interrupted agent runs; replaying stored traces where output arrays are empty; feeding the scorer custom input objects that omit inputMessages; filtering runs by date and including ones still in progress.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Invalid scoring filter: path(s) ${invalid.map(p => `"${p}"`)
• RUN_EXPERIMENT_FAILED_NO_DATA_PROVIDED
• Knowledge scope cannot be empty
• Input and output messages cannot be null or empty
• Agent response is required for prompt alignment scoring

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/a73c909291c8433b. Report an issue: GitHub.