Length of individual scores for '{metric}' should be the sam

Error message

Length of individual scores for '{metric}' should be the same as the inputs. Got {len(outputs['individual_scores'])} but expected {expected_len}.

What it means

The number of individual scores for a metric must equal the number of input examples (derived from the length of the first inputs list). A mismatch means the per-item scores cannot be aligned with inputs, so the constructor raises this ValueError with the got/expected counts.

Source

Thrown at haystack/evaluation/eval_run_result.py:58

        self.run_name = run_name
        self.inputs = deepcopy(inputs)
        self.results = deepcopy(results)

        if len(inputs) == 0:
            raise ValueError("No inputs provided.")
        if len({len(lst) for lst in inputs.values()}) != 1:
            raise ValueError("Lengths of the inputs should be the same.")

        expected_len = len(next(iter(inputs.values())))

        for metric, outputs in results.items():
            if "score" not in outputs:
                raise ValueError(f"Aggregate score missing for {metric}.")
            if "individual_scores" not in outputs:
                raise ValueError(f"Individual scores missing for {metric}.")

            if len(outputs["individual_scores"]) != expected_len:
                raise ValueError(
                    f"Length of individual scores for '{metric}' should be the same as the inputs. "
                    f"Got {len(outputs['individual_scores'])} but expected {expected_len}."
                )

    @staticmethod
    def _write_to_csv(csv_file: str, data: dict[str, list[Any]]) -> str:
        """
        Write data to a CSV file.

        :param csv_file: Path to the CSV file to write
        :param data: Dictionary containing the data to write
        :return: Status message indicating success or failure
        """
        list_lengths = [len(value) for value in data.values()]

        if len(set(list_lengths)) != 1:
            raise ValueError("All lists in the JSON must have the same length")

View on GitHub (pinned to e318778c9b)

Solutions

Emit exactly one score per input, even for failures (use None or 0 as placeholder).
Re-run evaluation with the same inputs used to build the result.
Trim or extend the individual_scores list to match the input count.
Pad with None: scores + [None] * (expected_len - len(scores)) if items were skipped.

Example fix

// before
inputs = {'queries': ['q1', 'q2']}
results = {'m': {'score': 0.33, 'individual_scores': [1, 0, 1]}}
// after
results = {'m': {'score': 0.5, 'individual_scores': [1, 0]}}

Defensive patterns

Strategy: validation

Validate before calling

expected = len(next(iter(inputs.values())))
for metric, outputs in results.items():
    assert len(outputs['individual_scores']) == expected, metric

Type guard

def scores_match_inputs(inputs: dict, outputs: dict) -> bool:
    expected = len(next(iter(inputs.values())))
    return isinstance(outputs.get('individual_scores'), list) and len(outputs['individual_scores']) == expected

Try / catch

try:
    run = EvaluationRunResult(run_name, inputs, results)
except ValueError as e:
    if 'Length of individual scores' in str(e):
        expected = len(next(iter(inputs.values())))
        for o in results.values():
            o['individual_scores'] = (o['individual_scores'] + [None] * expected)[:expected]
        run = EvaluationRunResult(run_name, inputs, results)
    else:
        raise

Prevention

Never filter inputs after computing scores
Emit one score per input even for failed items
Re-run evaluation rather than merging mismatched runs

When it happens

Trigger: Calling EvaluationRunResult(inputs, results) where len(results[metric]['individual_scores']) differs from len(next(iter(inputs.values()))) — e.g. 2 inputs but 3 individual_scores for a metric.

Common situations: Filtering inputs after evaluation but keeping full score lists; a metric skipping failed items instead of emitting a score per input; off-by-one errors in custom evaluation loops; merging partial runs incorrectly.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/df3779c339700db1. Report an issue: GitHub.