{"record":{"id":"fd2dafe18adc7dba","repo":"deepset-ai/haystack","slug":"comparative-scores-can-only-be-computed-between-ev","errorCode":null,"errorMessage":"Comparative scores can only be computed between EvaluationRunResults.","messagePattern":"Comparative scores can only be computed between EvaluationRunResults\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"haystack/evaluation/eval_run_result.py","lineNumber":189,"sourceCode":"    ) -> Union[str, \"DataFrame\", None]:\n        \"\"\"\n        Generates a report with detailed scores for each metric from two evaluation runs for comparison.\n\n        :param other: Results of another evaluation run to compare with.\n        :param keep_columns: List of common column names to keep from the inputs of the evaluation runs to compare.\n        :param output_format: The output format for the report, \"json\", \"csv\", or \"df\", default to \"json\".\n        :param csv_file: Filepath to save CSV output if `output_format` is \"csv\", must be provided.\n\n        :returns:\n            JSON or DataFrame with a comparison of the detailed scores, in case the output is set to a CSV file,\n             a message confirming the successful write or an error message.\n        :raises TypeError: If `other` is not an EvaluationRunResult instance, or if the detailed reports are not\n            dictionaries.\n        :raises ValueError: If the `other` parameter is missing required attributes.\n        \"\"\"\n\n        if not isinstance(other, EvaluationRunResult):\n            raise TypeError(\"Comparative scores can only be computed between EvaluationRunResults.\")\n\n        if not hasattr(other, \"run_name\") or not hasattr(other, \"inputs\") or not hasattr(other, \"results\"):\n            raise ValueError(\"The 'other' parameter must have 'run_name', 'inputs', and 'results' attributes.\")\n\n        if self.run_name == other.run_name:\n            logger.warning(\n                \"The run names of the two evaluation results are the same ('{run_name}')\", run_name=self.run_name\n            )\n\n        if self.inputs.keys() != other.inputs.keys():\n            logger.warning(\n                \"The input columns differ between the results; using the input columns of '{run_name}'\",\n                run_name=self.run_name,\n            )\n\n        # got both detailed reports\n        detailed_a = self.detailed_report(output_format=\"json\")\n        detailed_b = other.detailed_report(output_format=\"json\")","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/deepset-ai/haystack/blob/e318778c9bf60a1963e3b5f451359655dd696c30/haystack/evaluation/eval_run_result.py#L171-L207","documentation":"comparative_detailed_report compares two evaluation runs and requires the other object to be an EvaluationRunResult instance. Passing anything else (dict, None, another class) raises this TypeError, because the comparison logic depends on that class's run_name/inputs/results API.","triggerScenarios":"Calling run_a.comparative_detailed_report(other=some_dict) or other=None, or passing a differently-typed result object from another evaluation framework.","commonSituations":"Mixing results from two libraries; deserialized JSON results that were never re-wrapped in EvaluationRunResult; a refactor that changed the return type of a factory function.","solutions":["Wrap the comparison data in EvaluationRunResult before comparing: EvaluationRunResult(name, inputs, results).","Ensure both runs were produced by haystack's evaluation pipeline.","Check isinstance(other, EvaluationRunResult) before calling."],"exampleFix":"// before\nrun_a.comparative_detailed_report(other={'run_name': 'b', ...})\n// after\nrun_b = EvaluationRunResult('b', inputs_b, results_b)\nrun_a.comparative_detailed_report(other=run_b)","handlingStrategy":"type-guard","validationCode":"if not isinstance(other, EvaluationRunResult):\n    raise TypeError('other must be an EvaluationRunResult')","typeGuard":"def is_eval_run_result(obj: object) -> bool:\n    return isinstance(obj, EvaluationRunResult)","tryCatchPattern":"try:\n    comparison = run_a.comparative_detailed_report(other=other)\nexcept TypeError as e:\n    if 'EvaluationRunResults' in str(e):\n        other = EvaluationRunResult(other['run_name'], other['inputs'], other['results'])\n        comparison = run_a.comparative_detailed_report(other=other)\n    else:\n        raise","preventionTips":["Keep both runs as EvaluationRunResult instances end-to-end","Re-wrap deserialized results in EvaluationRunResult before comparing","Annotate function parameters with the EvaluationRunResult type"],"tags":["python","type-error","evaluation"],"backgroundTag":"wrong-argument-type","analyzedSha":"e318778c9bf60a1963e3b5f451359655dd696c30","analyzedAt":"2026-08-30T11:45:20.711Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}