deepset-ai/haystack · error

Comparative scores can only be computed between…

Error message

Comparative scores can only be computed between EvaluationRunResults.

What it means

comparative_detailed_report compares two evaluation runs and requires the other object to be an EvaluationRunResult instance. Passing anything else (dict, None, another class) raises this TypeError, because the comparison logic depends on that class's run_name/inputs/results API.

Solutions

1. Wrap the comparison data in EvaluationRunResult before comparing: EvaluationRunResult(name, inputs, results).
2. Ensure both runs were produced by haystack's evaluation pipeline.
3. Check isinstance(other, EvaluationRunResult) before calling.

Example fix

// before
run_a.comparative_detailed_report(other={'run_name': 'b', ...})
// after
run_b = EvaluationRunResult('b', inputs_b, results_b)
run_a.comparative_detailed_report(other=run_b)

if not isinstance(other, EvaluationRunResult):
    raise TypeError('other must be an EvaluationRunResult')

def is_eval_run_result(obj: object) -> bool:
    return isinstance(obj, EvaluationRunResult)

try:
    comparison = run_a.comparative_detailed_report(other=other)
except TypeError as e:
    if 'EvaluationRunResults' in str(e):
        other = EvaluationRunResult(other['run_name'], other['inputs'], other['results'])
        comparison = run_a.comparative_detailed_report(other=other)
    else:
        raise

When it happens

Trigger: Calling run_a.comparative_detailed_report(other=some_dict) or other=None, or passing a differently-typed result object from another evaluation framework.

Common situations: Mixing results from two libraries; deserialized JSON results that were never re-wrapped in EvaluationRunResult; a refactor that changed the return type of a factory function.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Detailed reports must be dictionaries.
• Aggregate score missing for
• All lists in the JSON must have the same length
• Expected ChatMessage object, got
• Individual scores missing for

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/fd2dafe18adc7dba. Report an issue: GitHub.

Appendix: source

Thrown at haystack/evaluation/eval_run_result.py:189

    ) -> Union[str, "DataFrame", None]:
        """
        Generates a report with detailed scores for each metric from two evaluation runs for comparison.

        :param other: Results of another evaluation run to compare with.
        :param keep_columns: List of common column names to keep from the inputs of the evaluation runs to compare.
        :param output_format: The output format for the report, "json", "csv", or "df", default to "json".
        :param csv_file: Filepath to save CSV output if `output_format` is "csv", must be provided.

        :returns:
            JSON or DataFrame with a comparison of the detailed scores, in case the output is set to a CSV file,
             a message confirming the successful write or an error message.
        :raises TypeError: If `other` is not an EvaluationRunResult instance, or if the detailed reports are not
            dictionaries.
        :raises ValueError: If the `other` parameter is missing required attributes.
        """

        if not isinstance(other, EvaluationRunResult):
            raise TypeError("Comparative scores can only be computed between EvaluationRunResults.")

        if not hasattr(other, "run_name") or not hasattr(other, "inputs") or not hasattr(other, "results"):
            raise ValueError("The 'other' parameter must have 'run_name', 'inputs', and 'results' attributes.")

        if self.run_name == other.run_name:
            logger.warning(
                "The run names of the two evaluation results are the same ('{run_name}')", run_name=self.run_name
            )

        if self.inputs.keys() != other.inputs.keys():
            logger.warning(
                "The input columns differ between the results; using the input columns of '{run_name}'",
                run_name=self.run_name,
            )

        # got both detailed reports
        detailed_a = self.detailed_report(output_format="json")
        detailed_b = other.detailed_report(output_format="json")

View on GitHub (pinned to e318778c9b)