The length of ground_truth_documents and retrieved_documents

Error message

The length of ground_truth_documents and retrieved_documents must be the same.

What it means

DocumentNDCGEvaluator.validate_inputs raises ValueError when ground_truth_documents and retrieved_documents have different lengths, since NDCG is computed per aligned question pair. This is the same alignment contract as MAP/MRR evaluators but enforced inside validate_inputs alongside other checks.

Source

Thrown at haystack/components/evaluators/document_ndcg.py:154

        Validate the input parameters.

        :param gt_docs:
            The ground_truth_documents to validate.
        :param ret_docs:
            The retrieved_documents to validate.

        :raises ValueError:
            If the ground_truth_documents or the retrieved_documents are an empty list.
            If the length of ground_truth_documents and retrieved_documents differs.
            If any list of documents in ground_truth_documents contains a mix of documents with and without a score.
        """
        if len(gt_docs) == 0 or len(ret_docs) == 0:
            msg = "ground_truth_documents and retrieved_documents must be provided."
            raise ValueError(msg)

        if len(gt_docs) != len(ret_docs):
            msg = "The length of ground_truth_documents and retrieved_documents must be the same."
            raise ValueError(msg)

        for docs in gt_docs:
            if any(doc.score is not None for doc in docs) and any(doc.score is None for doc in docs):
                msg = "Either none or all documents in each list of ground_truth_documents must have a score."
                raise ValueError(msg)

    def calculate_dcg(self, gt_docs: list[Document], ret_docs: list[Document]) -> float:
        """
        Calculate the discounted cumulative gain (DCG) of the retrieved documents.

        :param gt_docs:
            The ground truth documents.
        :param ret_docs:
            The retrieved documents.
        :returns:
            The discounted cumulative gain (DCG) of the retrieved
            documents based on the ground truth documents.
        """

View on GitHub (pinned to e318778c9b)

Solutions

Build both lists from the same iteration over question ids
Pad with [] or drop from both sides together so lengths match
Assert `len(gt) == len(ret)` before run()

Example fix

// before
ndcg.run(ground_truth_documents=gt[:5], retrieved_documents=retrieved)  # 5 vs 6
// after
n = min(len(gt), len(retrieved))
ndcg.run(ground_truth_documents=gt[:n], retrieved_documents=retrieved[:n])

Defensive patterns

Strategy: validation

Validate before calling

assert len(ground_truth_documents) == len(retrieved_documents), (len(ground_truth_documents), len(retrieved_documents))

Type guard

def aligned(a: list, b: list) -> bool:
    return len(a) == len(b)

Try / catch

try:
    result = evaluator.run(ground_truth_documents=gt, retrieved_documents=ret)
except ValueError as e:
    if "must be the same" in str(e):
        n = min(len(gt), len(ret)); result = evaluator.run(ground_truth_documents=gt[:n], retrieved_documents=ret[:n])
    else:
        raise

Prevention

Build both lists in the same loop over question ids
Never drop failed questions from only one side
Keep per-question results keyed by id so you can re-align
Sanity-check lengths before batch evaluation runs

When it happens

Trigger: Calling run() with, e.g., 5 ground-truth question lists and 6 retrieved question lists; any off-by-one from partial batch evaluation.

Common situations: Appending retrieved results in a loop that runs more/fewer iterations than the GT set; datasets where some questions failed and were dropped from one side only.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/bd2f557b850b63b2. Report an issue: GitHub.