linshenkx/prompt-optimizer · error · EvaluationParseError

Failed to parse evaluation result: no valid score JSON or re

Error message

Failed to parse evaluation result: no valid score JSON or recognizable overall score found. Raw content length: ${content.length} characters.

What it means

EvaluationParseError thrown after both structured JSON parsing and text-fallback parsing fail to find any score in the judge model's output. The service first tries to extract score JSON, then a text fallback (parseTextEvaluation); if neither yields a recognizable overall score, this error surfaces with the raw content length for diagnosis.

Source

Thrown at packages/core/src/services/evaluation/service.ts:2975

        const normalized = this.normalizeEvaluationResponse(payload as any, type, metadata);
        return normalized;
      } catch (e) {
        console.warn(
          '[EvaluationService] Failed to parse evaluation JSON candidate:',
          e instanceof Error ? e.message : String(e)
        );
      }
    }

    // 降级解析
    const textResult = this.parseTextEvaluation(content, type, metadata);
    if (textResult) {
      console.warn('[EvaluationService] Using text fallback parsing');
      return textResult;
    }

    throw new EvaluationParseError(
      `Failed to parse evaluation result: no valid score JSON or recognizable overall score found. Raw content length: ${content.length} characters.`
    );
  }

  /**
   * 从模型输出中提取可能的 JSON 片段。
   *
   * 现实中模型可能：
   * - 输出 ```json ... ```
   * - 输出 ``` ... ```（无语言标注）
   * - 在解释文字中夹杂一段 JSON
   */
  private extractJsonCandidates(content: string): string[] {
    const candidates: string[] = [];

    // 1) 优先提取所有 fenced code block（不限语言），只挑看起来像 JSON 的块。
    const fencedRegex = /```[a-zA-Z0-9_-]*\s*([\s\S]*?)\s*```/g;
    for (const match of content.matchAll(fencedRegex)) {

View on GitHub (pinned to 3e677b1d9f)

Solutions

1. Inspect the raw model content (log it where the error is caught) to see what was actually returned
2. Strengthen the judge prompt to require strict JSON with an overall score, or increase max_tokens to avoid truncation
3. Pin or switch to a judge model known to follow the format; retry once — occasional malformed outputs are common

Example fix

// before
const result = await svc.runEvaluation(req);

// after
let result;
try { result = await svc.runEvaluation(req); }
catch (e) {
  if (e instanceof EvaluationParseError) result = await svc.runEvaluation({ ...req, temperature: 0 });
  else throw e;
}

null

try { result = await svc.runEvaluation(req); }
catch (e) {
  if (e instanceof EvaluationParseError) {
    logRawContentForDebugging();
    result = await svc.runEvaluation({ ...req, judgePrompt: stricterJsonPrompt, temperature: 0 });
  } else throw e;
}

When it happens

Trigger: The judge model returns prose, refusal text, or malformed JSON with no parseable score field; or the output format drifted so neither the JSON extractor nor the regex/heuristic text parser recognizes an overall score.

Common situations: Switching judge models that ignore the output-format instructions; prompt-injection or safety refusals returning 'I cannot evaluate...'; truncated responses hitting max_tokens before the score is emitted; non-English outputs the text parser doesn't recognize.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Evaluation result is missing a valid overall score.
• Evaluation result is not a valid object.
• Evaluation result is missing the "score" field.
• Evaluation result is missing score for "${fieldName}".
• Invalid numeric score for "${fieldName}": ${value}

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/a51440a8257cbb97. Report an issue: GitHub.