The length of ground_truth_answers and predicted_answers mus

Error message

The length of ground_truth_answers and predicted_answers must be the same.

What it means

SASEvaluator.run (answer exact match) zips ground_truth_answers with predicted_answers using strict=True after asserting equal lengths. It raises ValueError when the two lists differ in length, since each predicted answer needs its ground-truth counterpart to compute the per-question match scores.

Source

Thrown at haystack/components/evaluators/answer_exact_match.py:57

    def run(self, ground_truth_answers: list[str], predicted_answers: list[str]) -> dict[str, Any]:
        """
        Run the AnswerExactMatchEvaluator on the given inputs.

        The `ground_truth_answers` and `retrieved_answers` must have the same length.

        :param ground_truth_answers:
            A list of expected answers.
        :param predicted_answers:
            A list of predicted answers.
        :returns:
            A dictionary with the following outputs:
            - `individual_scores` - A list of 0s and 1s, where 1 means that the predicted answer matched one of the
                ground truth.
            - `score` - A number from 0.0 to 1.0 that represents the proportion of questions where any predicted
                         answer matched one of the ground truth answers.
        """
        if not len(ground_truth_answers) == len(predicted_answers):
            raise ValueError("The length of ground_truth_answers and predicted_answers must be the same.")

        matches = []
        for truth, extracted in zip(ground_truth_answers, predicted_answers, strict=True):
            if truth == extracted:
                matches.append(1)
            else:
                matches.append(0)

        # The proportion of questions where any predicted answer matched one of the ground truth answers
        average = sum(matches) / len(predicted_answers)

        return {"individual_scores": matches, "score": average}

View on GitHub (pinned to e318778c9b)

Solutions

Ensure both lists are built from the same dataset rows with the same length
Align the lists by question id, padding or filtering consistently on both sides
Assert lengths match before calling run: `assert len(gt) == len(pred)`

Example fix

// before
evaluator.run(ground_truth_answers=["Paris", "Berlin"], predicted_answers=["Paris"])
// after
# keep answers from failed runs, e.g. as empty string
evaluator.run(ground_truth_answers=["Paris", "Berlin"], predicted_answers=["Paris", ""])

Defensive patterns

Strategy: validation

Validate before calling

assert len(ground_truth_answers) == len(predicted_answers), (len(ground_truth_answers), len(predicted_answers))

Type guard

def aligned(gt: list, pred: list) -> bool:
    return len(gt) == len(pred)

Try / catch

try:
    result = evaluator.run(ground_truth_answers=gt, predicted_answers=pred)
except ValueError as e:
    if "must be the same" in str(e):
        n = min(len(gt), len(pred)); result = evaluator.run(ground_truth_answers=gt[:n], predicted_answers=pred[:n])
    else:
        raise

Prevention

Iterate over the same dataset rows to build both lists
Keep failed predictions as empty strings instead of dropping rows
Align by question id before evaluation
Log list lengths before every evaluator.run call

When it happens

Trigger: Calling `evaluator.run(ground_truth_answers=["a", "b"], predicted_answers=["x"])` — any length mismatch between the two lists, including empty vs non-empty.

Common situations: Evaluation datasets where some questions have no model answer (LLM refused/failed) so the predictions list is shorter; loading answers from separate files or pipeline runs that dropped rows.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/bf4028fbfd0e713f. Report an issue: GitHub.