mastra-ai/mastra · error

Invalid relevance score returned by model: ${responseText}

Error message

Invalid relevance score returned by model: ${responseText}

What it means

The mastra-agent relevance scorer asks an LLM to output a 0-1 relevance score and parseRelevanceScore converts the text to a number. If the trimmed response is empty, not a finite number, or outside [0,1], this error is thrown with the raw model output for debugging.

Source

Thrown at packages/rag/src/rerank/relevance/mastra-agent/index.ts:11

import { Agent, isSupportedLanguageModel } from '@mastra/core/agent';
import type { MastraLanguageModel, MastraLegacyLanguageModel } from '@mastra/core/agent';
import { createSimilarityPrompt } from '@mastra/core/relevance';
import type { RelevanceScoreProvider } from '@mastra/core/relevance';

function parseRelevanceScore(responseText: string): number {
  const trimmed = responseText.trim();
  const score = Number(trimmed);

  if (!trimmed || !Number.isFinite(score) || score < 0 || score > 1) {
    throw new Error(`Invalid relevance score returned by model: ${responseText}`);
  }

  return score;
}

// Mastra Agent implementation
export class MastraAgentRelevanceScorer implements RelevanceScoreProvider {
  private agent: Agent;

  constructor(name: string, model: MastraLanguageModel | MastraLegacyLanguageModel) {
    this.agent = new Agent({
      id: `relevance-scorer-${name}`,
      name: `Relevance Scorer ${name}`,
      instructions: `You are a specialized agent for evaluating the relevance of text to queries.
Your task is to rate how well a text passage answers a given query.
Output only a number between 0 and 1, where:
1.0 = Perfectly relevant, directly answers the query
0.0 = Completely irrelevant

View on GitHub (pinned to 75dd419e61)

Solutions

1. Use a model that reliably follows format instructions, or lower temperature for deterministic numeric output
2. Ask for a single bare number in the scorer instructions and avoid extra prompt text
3. Catch the error and fall back to a default score or skip that document in reranking

Example fix

// before
const score = await scorer.getRelevanceScore(query, text); // throws on prose output
// after
let score;
try { score = await scorer.getRelevanceScore(query, text); }
catch { score = 0.5; }

// validate model output before use:
const parsed = Number(String(output).trim());
const usable = Number.isFinite(parsed) && parsed >= 0 && parsed <= 1;

const isScore = (v: unknown): v is number => typeof v === 'number' && Number.isFinite(v) && v >= 0 && v <= 1;

try { score = await scorer.getRelevanceScore(q, t); } catch (e) { if (e.message.includes('Invalid relevance score')) score = 0.5; else throw e; }

When it happens

Trigger: The LLM replies with prose like 'This document is quite relevant' instead of a bare number, replies in a format like '85%' or '0.85 (high)', or returns empty text.

Common situations: Weak/undersized model ignoring the score-only instruction; aggressive output constraints truncating the answer; prompt modified so the numeric-only instruction is lost.

Related errors

• No relevance score found on Cohere response
• Invalid state token payload
• Failed to parse A2A stream event: ${error instanceof Error ?
• @mastra/livekit: reply generation runs through the Mastra ag
• Unsupported GitHub review state: ${state}

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/8ebe3063552cee78. Report an issue: GitHub.