deepset-ai/haystack · error

Detailed reports must be dictionaries.

Error message

Detailed reports must be dictionaries.

What it means

The comparison calls detailed_report(output_format='json') on both runs and asserts the results are dictionaries. If either is not a dict (e.g. an overridden detailed_report returned a DataFrame or string), the merge logic cannot proceed and this TypeError is raised.

Source

Thrown at haystack/evaluation/eval_run_result.py:211

        if self.run_name == other.run_name:
            logger.warning(
                "The run names of the two evaluation results are the same ('{run_name}')", run_name=self.run_name
            )

        if self.inputs.keys() != other.inputs.keys():
            logger.warning(
                "The input columns differ between the results; using the input columns of '{run_name}'",
                run_name=self.run_name,
            )

        # got both detailed reports
        detailed_a = self.detailed_report(output_format="json")
        detailed_b = other.detailed_report(output_format="json")

        # ensure both detailed reports are in dictionaries format
        if not isinstance(detailed_a, dict) or not isinstance(detailed_b, dict):
            raise TypeError("Detailed reports must be dictionaries.")

        # determine which columns to ignore
        if keep_columns is None:
            ignore = list(self.inputs.keys())
        else:
            ignore = [col for col in list(self.inputs.keys()) if col not in keep_columns]

        # filter out ignored columns from pipe_b_dict
        filtered_detailed_b = {
            f"{other.run_name}_{key}": value for key, value in detailed_b.items() if key not in ignore
        }

        # rename columns in pipe_a_dict based on ignore list
        renamed_detailed_a = {
            (key if key in ignore else f"{self.run_name}_{key}"): value for key, value in detailed_a.items()
        }

        # combine both detailed reports

View on GitHub (pinned to e318778c9b)

Solutions

1. Keep detailed_report returning a dict for output_format='json'; do not override it to other types.
2. Call the base class implementation in an override, or fetch the dict directly.
3. Verify isinstance(run.detailed_report(output_format='json'), dict) before comparing.

Example fix

// before
class MyResult(EvaluationRunResult):
    def detailed_report(self, output_format='json'):
        return DataFrame(...)  # breaks comparison
// after
class MyResult(EvaluationRunResult):
    def detailed_report(self, output_format='json'):
        return super().detailed_report(output_format='json')  # dict for 'json'

if not isinstance(run_a.detailed_report(output_format='json'), dict):
    raise TypeError('detailed_report must return a dict for json format')

def returns_dict_report(run: EvaluationRunResult) -> bool:
    report = run.detailed_report(output_format='json')
    return isinstance(report, dict)

try:
    comparison = run_a.comparative_detailed_report(other=run_b)
except TypeError as e:
    if 'must be dictionaries' in str(e):
        raise TypeError('Override of detailed_report broke comparability; keep json output as dict') from e
    raise

When it happens

Trigger: Calling comparative_detailed_report on runs whose detailed_report was overridden or monkeypatched to return a non-dict format such as a pandas DataFrame or CSV string.

Common situations: Subclassing EvaluationRunResult and overriding detailed_report; monkeypatching detailed_report in tests to return other formats; future API changes altering the internal call.

Understand the failure class

Background: "Wrong argument type", "must be a string", "expected Array or Prism::Scope": TypeError and ArgumentError when a library receives a value of the wrong type — this error's family across 28 libraries.

Related errors

• Comparative scores can only be computed between EvaluationRu
• Unsupported source type {type(source)}
• meta must be either None, a dictionary or a list of dictiona
• MockDocumentEmbedder expects a list of Documents as input.In
• MockTextEmbedder expects a string as an input. In case you w

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/e4aa61d0653a36ee. Report an issue: GitHub.