deepset-ai/haystack · error

The number of predictions and labels must be the same.

Error message

The number of predictions and labels must be the same.

What it means

SASEvaluator computes semantic similarity pairwise between predicted answers and ground-truth answers, so the two lists must be the same length. It raises ValueError when the counts differ.

Source

Thrown at haystack/components/evaluators/sas_evaluator.py:145

    @component.output_types(score=float, individual_scores=list[float])
    def run(self, ground_truth_answers: list[str], predicted_answers: list[str]) -> dict[str, float | list[float]]:
        """
        SASEvaluator component run method.

        Run the SASEvaluator to compute the Semantic Answer Similarity (SAS) between a list of predicted answers
        and a list of ground truth answers. Both must be list of strings of same length.

        :param ground_truth_answers:
            A list of expected answers for each question.
        :param predicted_answers:
            A list of generated answers for each question.
        :returns:
            A dictionary with the following outputs:
                - `score`: Mean SAS score over all the predictions/ground-truth pairs.
                - `individual_scores`: A list of similarity scores for each prediction/ground-truth pair.
        """
        if len(ground_truth_answers) != len(predicted_answers):
            raise ValueError("The number of predictions and labels must be the same.")

        if any(answer is None for answer in predicted_answers):
            raise ValueError("Predicted answers must not contain None values.")

        if len(predicted_answers) == 0:
            return {"score": 0.0, "individual_scores": [0.0]}

        if not self._similarity_model:
            self.warm_up()

        if isinstance(self._similarity_model, CrossEncoder):
            # For Cross Encoders we create a list of pairs of predictions and labels
            sentence_pairs = list(zip(predicted_answers, ground_truth_answers, strict=True))
            similarity_scores = self._similarity_model.predict(
                sentence_pairs, batch_size=self._batch_size, convert_to_numpy=True
            )

            # All Cross Encoders do not return a set of logits scores that are normalized

View on GitHub (pinned to e318778c9b)

Solutions

1. Ensure ground_truth_answers and predicted_answers have identical lengths before calling run()
2. If invalid predictions must be removed, remove the corresponding ground-truth entries at the same indices (or pad)
3. Verify the upstream components' output lists were not sliced or filtered independently

Example fix

// before
evaluator.run(ground_truth_answers=gt, predicted_answers=preds[:5])
// after
evaluator.run(ground_truth_answers=gt[:5], predicted_answers=preds[:5])

if len(ground_truth_answers) != len(predicted_answers):
    raise ValueError(f"gt={len(ground_truth_answers)} preds={len(predicted_answers)}")

try:
    result = sas_evaluator.run(ground_truth_answers=gt, predicted_answers=preds)
except ValueError as e:
    logger.error("SAS input mismatch: %s", e)
    raise

When it happens

Trigger: Calling sas_evaluator.run(ground_truth_answers=[...], predicted_answers=[...]) with mismatched list lengths, e.g. 10 predictions against 10 labels but one list truncated or filtered.

Common situations: Dropping items from one list (e.g. removing empty predictions) without mirroring the other; different preprocessing pipelines applied to each list; passing questions instead of answers to one parameter.

Related errors

• LLM evaluator expects all input lists to have the same lengt
• Tool execution requires at least one tool.
• Number of replies ({len(replies)}), and metadata ({len(meta)
• The length of the metadata list must match the number of sou
• 'dimension' must be a positive integer.

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/bf34827c3ad454e1. Report an issue: GitHub.