deepset-ai/haystack · error

Predicted answers must not contain None values.

Error message

Predicted answers must not contain None values.

What it means

SASEvaluator cannot score None predicted answers, so it raises ValueError before invoking the similarity model to avoid None crashing the embedding step.

Source

Thrown at haystack/components/evaluators/sas_evaluator.py:148

        SASEvaluator component run method.

        Run the SASEvaluator to compute the Semantic Answer Similarity (SAS) between a list of predicted answers
        and a list of ground truth answers. Both must be list of strings of same length.

        :param ground_truth_answers:
            A list of expected answers for each question.
        :param predicted_answers:
            A list of generated answers for each question.
        :returns:
            A dictionary with the following outputs:
                - `score`: Mean SAS score over all the predictions/ground-truth pairs.
                - `individual_scores`: A list of similarity scores for each prediction/ground-truth pair.
        """
        if len(ground_truth_answers) != len(predicted_answers):
            raise ValueError("The number of predictions and labels must be the same.")

        if any(answer is None for answer in predicted_answers):
            raise ValueError("Predicted answers must not contain None values.")

        if len(predicted_answers) == 0:
            return {"score": 0.0, "individual_scores": [0.0]}

        if not self._similarity_model:
            self.warm_up()

        if isinstance(self._similarity_model, CrossEncoder):
            # For Cross Encoders we create a list of pairs of predictions and labels
            sentence_pairs = list(zip(predicted_answers, ground_truth_answers, strict=True))
            similarity_scores = self._similarity_model.predict(
                sentence_pairs, batch_size=self._batch_size, convert_to_numpy=True
            )

            # All Cross Encoders do not return a set of logits scores that are normalized
            # We normalize scores if they are larger than 1
            if (similarity_scores > 1).any():
                similarity_scores = expit(similarity_scores)

View on GitHub (pinned to e318778c9b)

Solutions

1. Filter or replace None values before evaluation: predicted_answers = [a if a is not None else "" for a in answers]
2. Fix the upstream component (e.g. use raise_on_failure=True on extractors/generators) so it never emits None
3. Validate with all(a is not None for a in predicted_answers) before calling run()

Example fix

// before
evaluator.run(ground_truth_answers=gt, predicted_answers=preds)
// after
cleaned = [p if p is not None else "" for p in preds]
evaluator.run(ground_truth_answers=gt, predicted_answers=cleaned)

if any(a is None for a in predicted_answers):
    predicted_answers = [a if a is not None else "" for a in predicted_answers]

def no_none_answers(answers: list) -> bool:
    return all(a is not None for a in answers)

try:
    result = sas_evaluator.run(ground_truth_answers=gt, predicted_answers=preds)
except ValueError:
    preds = [p or "" for p in preds]
    result = sas_evaluator.run(ground_truth_answers=gt, predicted_answers=preds)

When it happens

Trigger: Calling run() where predicted_answers contains at least one None element, e.g. an upstream extraction/generation component that returned None for a failed answer.

Common situations: Pipeline components that emit None on failure (e.g. extractors returning None content) feeding directly into the evaluator; JSON test data with null answers loaded from a file.

Related errors

• Tool execution requires at least one tool.
• 'dimension' must be a positive integer.
• 'dimension' must be a positive integer.
• LLM evaluator expected input parameter '{param}' but receive
• LLM evaluator expects all input values to be lists but recei

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/de2f353bb7e8a3fb. Report an issue: GitHub.