{"record":{"id":"e4aa61d0653a36ee","repo":"deepset-ai/haystack","slug":"detailed-reports-must-be-dictionaries","errorCode":null,"errorMessage":"Detailed reports must be dictionaries.","messagePattern":"Detailed reports must be dictionaries\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"haystack/evaluation/eval_run_result.py","lineNumber":211,"sourceCode":"\n        if self.run_name == other.run_name:\n            logger.warning(\n                \"The run names of the two evaluation results are the same ('{run_name}')\", run_name=self.run_name\n            )\n\n        if self.inputs.keys() != other.inputs.keys():\n            logger.warning(\n                \"The input columns differ between the results; using the input columns of '{run_name}'\",\n                run_name=self.run_name,\n            )\n\n        # got both detailed reports\n        detailed_a = self.detailed_report(output_format=\"json\")\n        detailed_b = other.detailed_report(output_format=\"json\")\n\n        # ensure both detailed reports are in dictionaries format\n        if not isinstance(detailed_a, dict) or not isinstance(detailed_b, dict):\n            raise TypeError(\"Detailed reports must be dictionaries.\")\n\n        # determine which columns to ignore\n        if keep_columns is None:\n            ignore = list(self.inputs.keys())\n        else:\n            ignore = [col for col in list(self.inputs.keys()) if col not in keep_columns]\n\n        # filter out ignored columns from pipe_b_dict\n        filtered_detailed_b = {\n            f\"{other.run_name}_{key}\": value for key, value in detailed_b.items() if key not in ignore\n        }\n\n        # rename columns in pipe_a_dict based on ignore list\n        renamed_detailed_a = {\n            (key if key in ignore else f\"{self.run_name}_{key}\"): value for key, value in detailed_a.items()\n        }\n\n        # combine both detailed reports","sourceCodeStart":193,"sourceCodeEnd":229,"githubUrl":"https://github.com/deepset-ai/haystack/blob/e318778c9bf60a1963e3b5f451359655dd696c30/haystack/evaluation/eval_run_result.py#L193-L229","documentation":"The comparison calls detailed_report(output_format='json') on both runs and asserts the results are dictionaries. If either is not a dict (e.g. an overridden detailed_report returned a DataFrame or string), the merge logic cannot proceed and this TypeError is raised.","triggerScenarios":"Calling comparative_detailed_report on runs whose detailed_report was overridden or monkeypatched to return a non-dict format such as a pandas DataFrame or CSV string.","commonSituations":"Subclassing EvaluationRunResult and overriding detailed_report; monkeypatching detailed_report in tests to return other formats; future API changes altering the internal call.","solutions":["Keep detailed_report returning a dict for output_format='json'; do not override it to other types.","Call the base class implementation in an override, or fetch the dict directly.","Verify isinstance(run.detailed_report(output_format='json'), dict) before comparing."],"exampleFix":"// before\nclass MyResult(EvaluationRunResult):\n    def detailed_report(self, output_format='json'):\n        return DataFrame(...)  # breaks comparison\n// after\nclass MyResult(EvaluationRunResult):\n    def detailed_report(self, output_format='json'):\n        return super().detailed_report(output_format='json')  # dict for 'json'","handlingStrategy":"type-guard","validationCode":"if not isinstance(run_a.detailed_report(output_format='json'), dict):\n    raise TypeError('detailed_report must return a dict for json format')","typeGuard":"def returns_dict_report(run: EvaluationRunResult) -> bool:\n    report = run.detailed_report(output_format='json')\n    return isinstance(report, dict)","tryCatchPattern":"try:\n    comparison = run_a.comparative_detailed_report(other=run_b)\nexcept TypeError as e:\n    if 'must be dictionaries' in str(e):\n        raise TypeError('Override of detailed_report broke comparability; keep json output as dict') from e\n    raise","preventionTips":["Do not override detailed_report to return non-dict types","Use composition instead of overriding when adding report formats","Test comparative_detailed_report on any EvaluationRunResult subclass"],"tags":["python","type-error","evaluation"],"backgroundTag":"wrong-argument-type","analyzedSha":"e318778c9bf60a1963e3b5f451359655dd696c30","analyzedAt":"2026-08-30T11:45:20.711Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}