deepset-ai/haystack · error

Aggregate score missing for {metric}.

Error message

Aggregate score missing for {metric}.

What it means

EvaluationRunResult validates that every metric entry in the results dict contains an aggregate 'score' key. If a metric's outputs dict lacks 'score', the constructor raises this ValueError immediately, refusing to build the result object. This guarantees all downstream reports (aggregated/detailed/comparative) can rely on an aggregate score existing per metric.

Source

Thrown at haystack/evaluation/eval_run_result.py:53

            Dictionary containing the results of the evaluators used in the evaluation pipeline. Each key is the name
            of the metric and its value is dictionary with the following keys:
                - 'score': The aggregated score for the metric.
                - 'individual_scores': A list of scores for each input sample.
        """
        self.run_name = run_name
        self.inputs = deepcopy(inputs)
        self.results = deepcopy(results)

        if len(inputs) == 0:
            raise ValueError("No inputs provided.")
        if len({len(lst) for lst in inputs.values()}) != 1:
            raise ValueError("Lengths of the inputs should be the same.")

        expected_len = len(next(iter(inputs.values())))

        for metric, outputs in results.items():
            if "score" not in outputs:
                raise ValueError(f"Aggregate score missing for {metric}.")
            if "individual_scores" not in outputs:
                raise ValueError(f"Individual scores missing for {metric}.")

            if len(outputs["individual_scores"]) != expected_len:
                raise ValueError(
                    f"Length of individual scores for '{metric}' should be the same as the inputs. "
                    f"Got {len(outputs['individual_scores'])} but expected {expected_len}."
                )

    @staticmethod
    def _write_to_csv(csv_file: str, data: dict[str, list[Any]]) -> str:
        """
        Write data to a CSV file.

        :param csv_file: Path to the CSV file to write
        :param data: Dictionary containing the data to write
        :return: Status message indicating success or failure
        """

View on GitHub (pinned to e318778c9b)

Solutions

  1. Ensure each metric's results dict contains an aggregate 'score' key, e.g. compute sum/mean of individual_scores.
  2. Use haystack's evaluate()/AggregateOutput format so results are built correctly.
  3. Validate results before constructing: check all(m.get('score') is not None for m in results.values()).

Example fix

// before
results = {'exact_match': {'individual_scores': [1, 0]}}
run = EvaluationRunResult('run', inputs, results)
// after
results = {'exact_match': {'score': 0.5, 'individual_scores': [1, 0]}}
run = EvaluationRunResult('run', inputs, results)
Defensive patterns

Strategy: validation

Validate before calling

for metric, outputs in results.items():
    if 'score' not in outputs:
        raise ValueError(f"Metric '{metric}' has no aggregate 'score'")

Type guard

def has_score(metric_output: dict) -> bool:
    return isinstance(metric_output, dict) and 'score' in metric_output

Try / catch

try:
    run = EvaluationRunResult(run_name, inputs, results)
except ValueError as e:
    if 'Aggregate score missing' in str(e):
        results = {m: {'score': 0.0, **o} for m, o in results.items()}
        run = EvaluationRunResult(run_name, inputs, results)
    else:
        raise

Prevention

When it happens

Trigger: Passing a results dict to EvaluationRunResult(...) where some metric maps to a dict without a 'score' key, e.g. {'metric_name': {'individual_scores': [...]}} missing the aggregate score produced by the metric callable.

Common situations: Hand-constructing results from a custom evaluation loop instead of haystack's evaluate() pipeline; a custom metric returning a partial result dict; renaming or dropping the 'score' key when post-processing saved evaluation output.

Understand the failure class

Background: "Missing required field" and "field is required" errors: why libraries reject payloads that omit mandatory fields — this error's family across 20 libraries.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/5e93c8b11609c888. Report an issue: GitHub.