What causes "Individual scores missing for {metric}." in deepset-ai/haystack?

Add an 'individual_scores' list to each metric's result dict with one score per input.

Individual scores missing for {metric}.

Error message

Individual scores missing for {metric}.

What it means

EvaluationRunResult requires each metric's outputs to include an 'individual_scores' list holding per-input scores. When that key is absent the constructor raises this ValueError. It exists so detailed and comparative reports always have per-example data available.

Source

Thrown at haystack/evaluation/eval_run_result.py:55

                - 'score': The aggregated score for the metric.
                - 'individual_scores': A list of scores for each input sample.
        """
        self.run_name = run_name
        self.inputs = deepcopy(inputs)
        self.results = deepcopy(results)

        if len(inputs) == 0:
            raise ValueError("No inputs provided.")
        if len({len(lst) for lst in inputs.values()}) != 1:
            raise ValueError("Lengths of the inputs should be the same.")

        expected_len = len(next(iter(inputs.values())))

        for metric, outputs in results.items():
            if "score" not in outputs:
                raise ValueError(f"Aggregate score missing for {metric}.")
            if "individual_scores" not in outputs:
                raise ValueError(f"Individual scores missing for {metric}.")

            if len(outputs["individual_scores"]) != expected_len:
                raise ValueError(
                    f"Length of individual scores for '{metric}' should be the same as the inputs. "
                    f"Got {len(outputs['individual_scores'])} but expected {expected_len}."
                )

    @staticmethod
    def _write_to_csv(csv_file: str, data: dict[str, list[Any]]) -> str:
        """
        Write data to a CSV file.

        :param csv_file: Path to the CSV file to write
        :param data: Dictionary containing the data to write
        :return: Status message indicating success or failure
        """
        list_lengths = [len(value) for value in data.values()]

View on GitHub (pinned to e318778c9b)

Solutions

Add an 'individual_scores' list to each metric's result dict with one score per input.
Make your metric callable return per-item scores, or compute them alongside the aggregate.
Pre-validate results: assert 'individual_scores' in metric_result for each metric.

Example fix

// before
results = {'exact_match': {'score': 0.5}}
// after
results = {'exact_match': {'score': 0.5, 'individual_scores': [1, 0]}}

Defensive patterns

Strategy: validation

Validate before calling

for metric, outputs in results.items():
    if 'individual_scores' not in outputs:
        raise ValueError(f"Metric '{metric}' has no 'individual_scores'")

Type guard

def has_individual_scores(metric_output: dict) -> bool:
    return isinstance(metric_output, dict) and isinstance(metric_output.get('individual_scores'), list)

Try / catch

try:
    run = EvaluationRunResult(run_name, inputs, results)
except ValueError as e:
    if 'Individual scores missing' in str(e):
        for m, o in results.items():
            o.setdefault('individual_scores', [None] * len(next(iter(inputs.values()))))
        run = EvaluationRunResult(run_name, inputs, results)
    else:
        raise

Prevention

Make every metric callable return per-item scores
Validate result dict keys after deserializing saved runs
Never hand-strip keys when post-processing results

When it happens

Trigger: Constructing EvaluationRunResult with a results dict like {'metric_name': {'score': 0.5}} where the per-input 'individual_scores' list was omitted.

Common situations: Custom metrics that only return an aggregate score; loading truncated evaluation output from JSON; hand-rolled evaluation pipelines that skip per-item scoring.

Understand the failure class

Background: "Missing required field" and "field is required" errors: why libraries reject payloads that omit mandatory fields — this error's family across 20 libraries.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/61583bf55396bffa. Report an issue: GitHub.