{"record":{"id":"bf34827c3ad454e1","repo":"deepset-ai/haystack","slug":"the-number-of-predictions-and-labels-must-be-the-s","errorCode":null,"errorMessage":"The number of predictions and labels must be the same.","messagePattern":"The number of predictions and labels must be the same\\.","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"haystack/components/evaluators/sas_evaluator.py","lineNumber":145,"sourceCode":"    @component.output_types(score=float, individual_scores=list[float])\n    def run(self, ground_truth_answers: list[str], predicted_answers: list[str]) -> dict[str, float | list[float]]:\n        \"\"\"\n        SASEvaluator component run method.\n\n        Run the SASEvaluator to compute the Semantic Answer Similarity (SAS) between a list of predicted answers\n        and a list of ground truth answers. Both must be list of strings of same length.\n\n        :param ground_truth_answers:\n            A list of expected answers for each question.\n        :param predicted_answers:\n            A list of generated answers for each question.\n        :returns:\n            A dictionary with the following outputs:\n                - `score`: Mean SAS score over all the predictions/ground-truth pairs.\n                - `individual_scores`: A list of similarity scores for each prediction/ground-truth pair.\n        \"\"\"\n        if len(ground_truth_answers) != len(predicted_answers):\n            raise ValueError(\"The number of predictions and labels must be the same.\")\n\n        if any(answer is None for answer in predicted_answers):\n            raise ValueError(\"Predicted answers must not contain None values.\")\n\n        if len(predicted_answers) == 0:\n            return {\"score\": 0.0, \"individual_scores\": [0.0]}\n\n        if not self._similarity_model:\n            self.warm_up()\n\n        if isinstance(self._similarity_model, CrossEncoder):\n            # For Cross Encoders we create a list of pairs of predictions and labels\n            sentence_pairs = list(zip(predicted_answers, ground_truth_answers, strict=True))\n            similarity_scores = self._similarity_model.predict(\n                sentence_pairs, batch_size=self._batch_size, convert_to_numpy=True\n            )\n\n            # All Cross Encoders do not return a set of logits scores that are normalized","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/deepset-ai/haystack/blob/e318778c9bf60a1963e3b5f451359655dd696c30/haystack/components/evaluators/sas_evaluator.py#L127-L163","documentation":"SASEvaluator computes semantic similarity pairwise between predicted answers and ground-truth answers, so the two lists must be the same length. It raises ValueError when the counts differ.","triggerScenarios":"Calling sas_evaluator.run(ground_truth_answers=[...], predicted_answers=[...]) with mismatched list lengths, e.g. 10 predictions against 10 labels but one list truncated or filtered.","commonSituations":"Dropping items from one list (e.g. removing empty predictions) without mirroring the other; different preprocessing pipelines applied to each list; passing questions instead of answers to one parameter.","solutions":["Ensure ground_truth_answers and predicted_answers have identical lengths before calling run()","If invalid predictions must be removed, remove the corresponding ground-truth entries at the same indices (or pad)","Verify the upstream components' output lists were not sliced or filtered independently"],"exampleFix":"// before\nevaluator.run(ground_truth_answers=gt, predicted_answers=preds[:5])\n// after\nevaluator.run(ground_truth_answers=gt[:5], predicted_answers=preds[:5])","handlingStrategy":"validation","validationCode":"if len(ground_truth_answers) != len(predicted_answers):\n    raise ValueError(f\"gt={len(ground_truth_answers)} preds={len(predicted_answers)}\")","typeGuard":null,"tryCatchPattern":"try:\n    result = sas_evaluator.run(ground_truth_answers=gt, predicted_answers=preds)\nexcept ValueError as e:\n    logger.error(\"SAS input mismatch: %s\", e)\n    raise","preventionTips":["Build gt/preds pairs with zip(...) so they cannot diverge","Never filter one list without the other","Assert equality of lengths in evaluation test fixtures"],"tags":["python","validation","length-mismatch"],"backgroundTag":"list-length-mismatch","analyzedSha":"e318778c9bf60a1963e3b5f451359655dd696c30","analyzedAt":"2026-08-30T11:45:20.711Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}