The length of ground_truth_documents and retrieved_documents

Error message

The length of ground_truth_documents and retrieved_documents must be the same.

What it means

DocumentMAPEvaluator.run requires ground_truth_documents and retrieved_documents to be parallel lists of per-question document lists. It raises ValueError when their lengths differ, because average precision must be computed per question pair and a strict zip would silently drop or misalign questions.

Source

Thrown at haystack/components/evaluators/document_map.py:113

    ) -> dict[str, Any]:
        """
        Run the DocumentMAPEvaluator on the given inputs.

        All lists must have the same length.

        :param ground_truth_documents:
            A list of expected documents for each question.
        :param retrieved_documents:
            A list of retrieved documents for each question.
        :returns:
            A dictionary with the following outputs:
            - `score` - The average of calculated scores.
            - `individual_scores` - A list of numbers from 0.0 to 1.0 that represents how high retrieved documents
                are ranked.
        """
        if len(ground_truth_documents) != len(retrieved_documents):
            msg = "The length of ground_truth_documents and retrieved_documents must be the same."
            raise ValueError(msg)

        individual_scores = []

        for ground_truth, retrieved in zip(ground_truth_documents, retrieved_documents, strict=True):
            average_precision = 0.0
            average_precision_numerator = 0.0
            retrieved_relevant_documents = 0

            # A list keeps the deduplication working for unhashable comparison values, for example when
            # document_comparison_field points to a meta key holding a list.
            uncredited_ground_truth_values: list[Any] = []
            for doc in ground_truth:
                value = self._get_comparison_value(doc)
                if value is not None and value not in uncredited_ground_truth_values:
                    uncredited_ground_truth_values.append(value)

            total_relevant_documents = len(uncredited_ground_truth_values)
            for rank, retrieved_document in enumerate(retrieved):

View on GitHub (pinned to e318778c9b)

Solutions

Make both lists contain one entry per question, using empty lists [] for questions with no documents
Filter both lists together (same indices) before calling run
Validate `len(a) == len(b)` in your eval harness before invoking the evaluator

Example fix

// before
evaluator.run(ground_truth_documents=gt[:9], retrieved_documents=retrieved)  # 9 vs 10
// after
evaluator.run(ground_truth_documents=gt, retrieved_documents=retrieved)  # keep aligned, use [] where empty

Defensive patterns

Strategy: validation

Validate before calling

if len(ground_truth_documents) != len(retrieved_documents):
    raise ValueError(f"gt={len(ground_truth_documents)} retrieved={len(retrieved_documents)}")

Type guard

def aligned(a: list, b: list) -> bool:
    return len(a) == len(b)

Try / catch

try:
    result = evaluator.run(ground_truth_documents=gt, retrieved_documents=ret)
except ValueError as e:
    if "must be the same" in str(e):
        n = min(len(gt), len(ret)); result = evaluator.run(ground_truth_documents=gt[:n], retrieved_documents=ret[:n])
    else:
        raise

Prevention

Emit one entry per question even when documents are missing (use [])
Build both lists from a single loop over the dataset
Keep retrieval failures as empty sub-lists, not missing rows
Assert lengths in the evaluation harness

When it happens

Trigger: Calling run() where `len(ground_truth_documents) != len(retrieved_documents)`, e.g. 10 ground-truth entries but only 9 retrieval results after a failed retrieval.

Common situations: Building evaluation sets from separate files where a query was skipped; retriever returning results only for successful queries; concatenating batch results unevenly.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/1e2f0c2f7c10bc45. Report an issue: GitHub.