{"record":{"id":"8ebe3063552cee78","repo":"mastra-ai/mastra","slug":"invalid-relevance-score-returned-by-model-respo","errorCode":null,"errorMessage":"Invalid relevance score returned by model: ${responseText}","messagePattern":"Invalid relevance score returned by model: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/rag/src/rerank/relevance/mastra-agent/index.ts","lineNumber":11,"sourceCode":"import { Agent, isSupportedLanguageModel } from '@mastra/core/agent';\nimport type { MastraLanguageModel, MastraLegacyLanguageModel } from '@mastra/core/agent';\nimport { createSimilarityPrompt } from '@mastra/core/relevance';\nimport type { RelevanceScoreProvider } from '@mastra/core/relevance';\n\nfunction parseRelevanceScore(responseText: string): number {\n  const trimmed = responseText.trim();\n  const score = Number(trimmed);\n\n  if (!trimmed || !Number.isFinite(score) || score < 0 || score > 1) {\n    throw new Error(`Invalid relevance score returned by model: ${responseText}`);\n  }\n\n  return score;\n}\n\n// Mastra Agent implementation\nexport class MastraAgentRelevanceScorer implements RelevanceScoreProvider {\n  private agent: Agent;\n\n  constructor(name: string, model: MastraLanguageModel | MastraLegacyLanguageModel) {\n    this.agent = new Agent({\n      id: `relevance-scorer-${name}`,\n      name: `Relevance Scorer ${name}`,\n      instructions: `You are a specialized agent for evaluating the relevance of text to queries.\nYour task is to rate how well a text passage answers a given query.\nOutput only a number between 0 and 1, where:\n1.0 = Perfectly relevant, directly answers the query\n0.0 = Completely irrelevant","sourceCodeStart":1,"sourceCodeEnd":29,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/rag/src/rerank/relevance/mastra-agent/index.ts#L1-L29","documentation":"The mastra-agent relevance scorer asks an LLM to output a 0-1 relevance score and parseRelevanceScore converts the text to a number. If the trimmed response is empty, not a finite number, or outside [0,1], this error is thrown with the raw model output for debugging.","triggerScenarios":"The LLM replies with prose like 'This document is quite relevant' instead of a bare number, replies in a format like '85%' or '0.85 (high)', or returns empty text.","commonSituations":"Weak/undersized model ignoring the score-only instruction; aggressive output constraints truncating the answer; prompt modified so the numeric-only instruction is lost.","solutions":["Use a model that reliably follows format instructions, or lower temperature for deterministic numeric output","Ask for a single bare number in the scorer instructions and avoid extra prompt text","Catch the error and fall back to a default score or skip that document in reranking"],"exampleFix":"// before\nconst score = await scorer.getRelevanceScore(query, text); // throws on prose output\n// after\nlet score;\ntry { score = await scorer.getRelevanceScore(query, text); }\ncatch { score = 0.5; }","handlingStrategy":"fallback","validationCode":"// validate model output before use:\nconst parsed = Number(String(output).trim());\nconst usable = Number.isFinite(parsed) && parsed >= 0 && parsed <= 1;","typeGuard":"const isScore = (v: unknown): v is number => typeof v === 'number' && Number.isFinite(v) && v >= 0 && v <= 1;","tryCatchPattern":"try { score = await scorer.getRelevanceScore(q, t); } catch (e) { if (e.message.includes('Invalid relevance score')) score = 0.5; else throw e; }","preventionTips":["Use low temperature and explicit 'reply with a single number between 0 and 1' instructions","Prefer structured output / JSON mode when the model supports it","Retry once on parse failure before falling back to a neutral score"],"tags":["llm","parsing","rerank","response-parsing"],"backgroundTag":"llm-output-parse-failure","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}