mastra-ai/mastra · error · Error

Either expectedTool or expectedToolOrder must be provided

Error message

Either expectedTool or expectedToolOrder must be provided

What it means

createToolCallAccuracyScorerCode requires configuration describing what to expect: either a single expectedTool name or an expectedToolOrder array. If neither option is provided there is nothing to compare the run's tool calls against, so the factory throws at construction time instead of producing a scorer that can never pass. This is a factory-time configuration validation, not a runtime scoring failure.

Source

Thrown at packages/evals/src/scorers/code/tool-call-accuracy/index.ts:68

    return checkToolOrder(actualTools, expectedToolOrder, strictMode) ? 1 : 0;
  }

  if (!expectedTool) {
    return 0;
  }

  if (strictMode) {
    return actualTools.length === 1 && actualTools[0] === expectedTool ? 1 : 0;
  }

  return actualTools.includes(expectedTool) ? 1 : 0;
}

export function createToolCallAccuracyScorerCode(options: ToolCallAccuracyOptions) {
  const { expectedTool, strictMode = false, expectedToolOrder } = options;

  if (!expectedTool && !expectedToolOrder) {
    throw new Error('Either expectedTool or expectedToolOrder must be provided');
  }

  const getDescription = () => {
    return expectedToolOrder
      ? `Evaluates whether the LLM called tools in the correct order: [${expectedToolOrder.join(', ')}]`
      : `Evaluates whether the LLM selected the correct tool (${expectedTool}) from the available tools`;
  };

  return createScorer({
    id: 'code-tool-call-accuracy-scorer',
    name: 'Tool Call Accuracy Scorer',
    description: getDescription(),
    type: 'agent',
  })
    .preprocess(async ({ run }) => {
      const isInputInvalid = !run.input || !run.input.inputMessages || run.input.inputMessages.length === 0;
      const isOutputInvalid = !run.output || run.output.length === 0;

View on GitHub (pinned to 75dd419e61)

Solutions

1. Pass expectedTool: 'toolName' when you only care whether a specific tool was called
2. Pass expectedToolOrder: ['toolA','toolB'] when the sequence of tool calls matters
3. Verify the option keys are spelled expectedTool and expectedToolOrder exactly
4. If you want appropriateness without a fixed expectation, use the appropriate scorer variant (toolCallAppropriatenessScorer defaults) instead

Example fix

// before
const scorer = createToolCallAccuracyScorerCode({});

// after
const scorer = createToolCallAccuracyScorerCode({ expectedTool: 'getWeather' });
// or
const scorer = createToolCallAccuracyScorerCode({ expectedToolOrder: ['search', 'summarize'] });

function validateToolAccuracyOptions(o: ToolCallAccuracyOptions): void {
  if (!o.expectedTool && !o.expectedToolOrder) {
    throw new Error('Provide expectedTool or expectedToolOrder');
  }
}
validateToolAccuracyOptions(options);

function hasExpectation(o: ToolCallAccuracyOptions): o is ToolCallAccuracyOptions & ({ expectedTool: string } | { expectedToolOrder: string[] }) {
  return Boolean(o.expectedTool) || Boolean(o.expectedToolOrder);
}

try {
  const scorer = createToolCallAccuracyScorerCode(options);
} catch (err) {
  if ((err as Error).message.includes('expectedTool or expectedToolOrder')) {
    throw new ConfigError('Tool-call accuracy scorer requires expectedTool or expectedToolOrder');
  }
  throw err;
}

When it happens

Trigger: Calling createToolCallAccuracyScorerCode({}) or createToolCallAccuracyScorerCode({ strictMode: true }) without expectedTool or expectedToolOrder; building the scorer programmatically from config where both fields are undefined.

Common situations: Typo in the options key (expectedTools vs expectedTool); loading scorer options from JSON/env where the fields are omitted; copying a scorer factory call and deleting the expectedTool line.

Related errors

• MASTR_SCORER_FAILED_TO_CREATE_MISSING_ID
• Input and output messages cannot be null or empty
• Either context or contextExtractor is required for Context P
• Either context or contextExtractor is required for Context R
• MASTR_SCORER_FAILED_TO_RUN_MISSING_GENERATE_SCORE

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/f716e4ef77d52649. Report an issue: GitHub.