The length of ground_truth_documents and retrieved_documents

Error message

The length of ground_truth_documents and retrieved_documents must be the same.

What it means

DocumentMRREvaluator.run requires ground_truth_documents and retrieved_documents to be equal-length parallel lists of per-question document lists, computing reciprocal rank per question. It raises ValueError on length mismatch because strict per-question pairing is required.

Source

Thrown at haystack/components/evaluators/document_mrr.py:111

    ) -> dict[str, Any]:
        """
        Run the DocumentMRREvaluator on the given inputs.

        `ground_truth_documents` and `retrieved_documents` must have the same length.

        :param ground_truth_documents:
            A list of expected documents for each question.
        :param retrieved_documents:
            A list of retrieved documents for each question.
        :returns:
            A dictionary with the following outputs:
            - `score` - The average of calculated scores.
            - `individual_scores` - A list of numbers from 0.0 to 1.0 that represents how high the first retrieved
                document is ranked.
        """
        if len(ground_truth_documents) != len(retrieved_documents):
            msg = "The length of ground_truth_documents and retrieved_documents must be the same."
            raise ValueError(msg)

        individual_scores = []

        for ground_truth, retrieved in zip(ground_truth_documents, retrieved_documents, strict=True):
            reciprocal_rank = 0.0

            ground_truth_values = [val for doc in ground_truth if (val := self._get_comparison_value(doc)) is not None]
            for rank, retrieved_document in enumerate(retrieved):
                retrieved_value = self._get_comparison_value(retrieved_document)
                if retrieved_value is None:
                    continue
                if retrieved_value in ground_truth_values:
                    reciprocal_rank = 1 / (rank + 1)
                    break
            individual_scores.append(reciprocal_rank)

        score = sum(individual_scores) / len(ground_truth_documents)

View on GitHub (pinned to e318778c9b)

Solutions

Keep both lists one-entry-per-question, using [] for missing document sets
Filter/pad both lists together by question id
Add a pre-run length assertion in your evaluation script

Example fix

// before
mrr.run(ground_truth_documents=gt, retrieved_documents=retrieved[1:])  # dropped first row
// after
mrr.run(ground_truth_documents=gt, retrieved_documents=retrieved)  # both length N

Defensive patterns

Strategy: validation

Validate before calling

assert len(ground_truth_documents) == len(retrieved_documents), (len(ground_truth_documents), len(retrieved_documents))

Type guard

def aligned(a: list, b: list) -> bool:
    return len(a) == len(b)

Try / catch

try:
    result = evaluator.run(ground_truth_documents=gt, retrieved_documents=ret)
except ValueError as e:
    if "must be the same" in str(e):
        n = min(len(gt), len(ret)); result = evaluator.run(ground_truth_documents=gt[:n], retrieved_documents=ret[:n])
    else:
        raise

Prevention

Keep GT and retrieved lists indexed by the same question ids
Use [] for questions without documents rather than omitting rows
Slice both lists together when subsetting
Validate lengths before running large evaluation sweeps

When it happens

Trigger: Calling run() with different numbers of ground-truth question entries vs retrieval result entries, e.g. one list has 20 items and the other 19 after a failed query.

Common situations: Evaluation harnesses that skip queries with no retrieval output; loading GT and predictions from independent JSONL files with diverging row counts.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/20bb91f1f4aee6c9. Report an issue: GitHub.