{"record":{"id":"df3779c339700db1","repo":"deepset-ai/haystack","slug":"length-of-individual-scores-for-metric-should","errorCode":null,"errorMessage":"Length of individual scores for '{metric}' should be the same as the inputs. Got {len(outputs['individual_scores'])} but expected {expected_len}.","messagePattern":"Length of individual scores for '(.+?)' should be the same as the inputs\\. Got (.+?) but expected (.+?)\\.","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"haystack/evaluation/eval_run_result.py","lineNumber":58,"sourceCode":"        self.run_name = run_name\n        self.inputs = deepcopy(inputs)\n        self.results = deepcopy(results)\n\n        if len(inputs) == 0:\n            raise ValueError(\"No inputs provided.\")\n        if len({len(lst) for lst in inputs.values()}) != 1:\n            raise ValueError(\"Lengths of the inputs should be the same.\")\n\n        expected_len = len(next(iter(inputs.values())))\n\n        for metric, outputs in results.items():\n            if \"score\" not in outputs:\n                raise ValueError(f\"Aggregate score missing for {metric}.\")\n            if \"individual_scores\" not in outputs:\n                raise ValueError(f\"Individual scores missing for {metric}.\")\n\n            if len(outputs[\"individual_scores\"]) != expected_len:\n                raise ValueError(\n                    f\"Length of individual scores for '{metric}' should be the same as the inputs. \"\n                    f\"Got {len(outputs['individual_scores'])} but expected {expected_len}.\"\n                )\n\n    @staticmethod\n    def _write_to_csv(csv_file: str, data: dict[str, list[Any]]) -> str:\n        \"\"\"\n        Write data to a CSV file.\n\n        :param csv_file: Path to the CSV file to write\n        :param data: Dictionary containing the data to write\n        :return: Status message indicating success or failure\n        \"\"\"\n        list_lengths = [len(value) for value in data.values()]\n\n        if len(set(list_lengths)) != 1:\n            raise ValueError(\"All lists in the JSON must have the same length\")\n","sourceCodeStart":40,"sourceCodeEnd":76,"githubUrl":"https://github.com/deepset-ai/haystack/blob/e318778c9bf60a1963e3b5f451359655dd696c30/haystack/evaluation/eval_run_result.py#L40-L76","documentation":"The number of individual scores for a metric must equal the number of input examples (derived from the length of the first inputs list). A mismatch means the per-item scores cannot be aligned with inputs, so the constructor raises this ValueError with the got/expected counts.","triggerScenarios":"Calling EvaluationRunResult(inputs, results) where len(results[metric]['individual_scores']) differs from len(next(iter(inputs.values()))) — e.g. 2 inputs but 3 individual_scores for a metric.","commonSituations":"Filtering inputs after evaluation but keeping full score lists; a metric skipping failed items instead of emitting a score per input; off-by-one errors in custom evaluation loops; merging partial runs incorrectly.","solutions":["Emit exactly one score per input, even for failures (use None or 0 as placeholder).","Re-run evaluation with the same inputs used to build the result.","Trim or extend the individual_scores list to match the input count.","Pad with None: scores + [None] * (expected_len - len(scores)) if items were skipped."],"exampleFix":"// before\ninputs = {'queries': ['q1', 'q2']}\nresults = {'m': {'score': 0.33, 'individual_scores': [1, 0, 1]}}\n// after\nresults = {'m': {'score': 0.5, 'individual_scores': [1, 0]}}","handlingStrategy":"validation","validationCode":"expected = len(next(iter(inputs.values())))\nfor metric, outputs in results.items():\n    assert len(outputs['individual_scores']) == expected, metric","typeGuard":"def scores_match_inputs(inputs: dict, outputs: dict) -> bool:\n    expected = len(next(iter(inputs.values())))\n    return isinstance(outputs.get('individual_scores'), list) and len(outputs['individual_scores']) == expected","tryCatchPattern":"try:\n    run = EvaluationRunResult(run_name, inputs, results)\nexcept ValueError as e:\n    if 'Length of individual scores' in str(e):\n        expected = len(next(iter(inputs.values())))\n        for o in results.values():\n            o['individual_scores'] = (o['individual_scores'] + [None] * expected)[:expected]\n        run = EvaluationRunResult(run_name, inputs, results)\n    else:\n        raise","preventionTips":["Never filter inputs after computing scores","Emit one score per input even for failed items","Re-run evaluation rather than merging mismatched runs"],"tags":["python","validation","evaluation"],"backgroundTag":"length-mismatch","analyzedSha":"e318778c9bf60a1963e3b5f451359655dd696c30","analyzedAt":"2026-08-30T11:45:20.711Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}