linshenkx/prompt-optimizer · error · EvaluationParseError

Evaluation result is missing a valid overall score.

Error message

Evaluation result is missing a valid overall score.

What it means

Thrown by EvaluationService when parsing an evaluation result yields no valid overall score (overall is null). The service can synthesize a minimal dimensions array from overall, but if overall itself is missing it cannot construct an EvaluationScore and aborts with EvaluationParseError.

Solutions

1. Inspect the raw evaluation payload and confirm the overall score field name/format expected by the parser
2. Update the evaluation prompt to explicitly require a numeric overall score in the response schema
3. Add pre-validation of the model response before passing it to EvaluationService
4. If dimensions exist but overall is absent, compute overall yourself and pass it in

Example fix

// before
const result = await evaluationService.evaluate(rawModelOutput);
// after
const parsed = JSON.parse(rawModelOutput);
if (typeof parsed.overall !== 'number') throw new Error('model output lacks overall');
const result = await evaluationService.evaluate(rawModelOutput);

function hasOverall(raw: unknown): boolean {
  const o = typeof raw === 'string' ? JSON.parse(raw) : raw;
  return o != null && typeof (o as any).overall === 'number' && Number.isFinite((o as any).overall);
}

const isEvalPayload = (p: unknown): p is { overall: number } =>
  typeof p === 'object' && p !== null && typeof (p as any).overall === 'number';

try { await svc.evaluate(raw); } catch (e) { if (e instanceof EvaluationParseError) { /* re-prompt model or fallback */ } throw e; }

When it happens

Trigger: Calling EvaluationService's parse/evaluation API with a model response or JSON payload that contains no parseable overall score field (or a non-numeric one), e.g. an LLM returning only per-dimension scores or malformed JSON.

Common situations: LLM output format drift after a prompt/model change; evaluation prompt not instructing the model to emit an overall score; locale-specific number formats that fail numeric parsing; truncated responses.

Related errors

• Failed to parse evaluation result: no valid score JSON or…
• Compare evaluation requires at least one test case.
• Compare evaluation requires at least two snapshots.
• Compare snapshot # references unknown testCaseId " ".
• Compare test case # id must be unique.

AI-assisted analysis of linshenkx/prompt-optimizer@3e677b1d9f (2026-08-27). Data as JSON: /api/errors/a0d463e14c426e12. Report an issue: GitHub.

Appendix: source

Thrown at packages/core/src/services/evaluation/service.ts:3232

        }
      }
    }

    // 如果 overall 缺失，但维度存在，则按平均分计算。
    if (overall === null && dimensions.length > 0) {
      const avg = Math.round(
        dimensions.reduce((sum, d) => sum + d.score, 0) / dimensions.length
      );
      overall = Math.max(0, Math.min(100, avg));
    }

    // 如果维度缺失，但 overall 存在，则返回一个最小维度数组。
    if (dimensions.length === 0 && overall !== null) {
      dimensions = [{ key: 'overall', label: '综合评分', score: overall }];
    }

    if (overall === null) {
      throw new EvaluationParseError('Evaluation result is missing a valid overall score.');
    }

    const score: EvaluationScore = {
      overall,
      dimensions,
    };

    // 解析 improvements（最多3条）
    const improvements = Array.isArray(data.improvements)
      ? data.improvements.map((x: any) => String(x)).filter(Boolean).slice(0, 3)
      : typeof data.improvements === 'string' && data.improvements.trim()
        ? [data.improvements.trim()].slice(0, 3)
        : [];

    // 解析 patchPlan（最多3条）
    const patchPlan = this.normalizePatchPlan(data.patchPlan || []).slice(0, 3);

    const summary = typeof data.summary === 'string' ? data.summary : '';

View on GitHub (pinned to 3e677b1d9f)