{"record":{"id":"61583bf55396bffa","repo":"deepset-ai/haystack","slug":"individual-scores-missing-for-metric","errorCode":null,"errorMessage":"Individual scores missing for {metric}.","messagePattern":"Individual scores missing for (.+?)\\.","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"haystack/evaluation/eval_run_result.py","lineNumber":55,"sourceCode":"                - 'score': The aggregated score for the metric.\n                - 'individual_scores': A list of scores for each input sample.\n        \"\"\"\n        self.run_name = run_name\n        self.inputs = deepcopy(inputs)\n        self.results = deepcopy(results)\n\n        if len(inputs) == 0:\n            raise ValueError(\"No inputs provided.\")\n        if len({len(lst) for lst in inputs.values()}) != 1:\n            raise ValueError(\"Lengths of the inputs should be the same.\")\n\n        expected_len = len(next(iter(inputs.values())))\n\n        for metric, outputs in results.items():\n            if \"score\" not in outputs:\n                raise ValueError(f\"Aggregate score missing for {metric}.\")\n            if \"individual_scores\" not in outputs:\n                raise ValueError(f\"Individual scores missing for {metric}.\")\n\n            if len(outputs[\"individual_scores\"]) != expected_len:\n                raise ValueError(\n                    f\"Length of individual scores for '{metric}' should be the same as the inputs. \"\n                    f\"Got {len(outputs['individual_scores'])} but expected {expected_len}.\"\n                )\n\n    @staticmethod\n    def _write_to_csv(csv_file: str, data: dict[str, list[Any]]) -> str:\n        \"\"\"\n        Write data to a CSV file.\n\n        :param csv_file: Path to the CSV file to write\n        :param data: Dictionary containing the data to write\n        :return: Status message indicating success or failure\n        \"\"\"\n        list_lengths = [len(value) for value in data.values()]\n","sourceCodeStart":37,"sourceCodeEnd":73,"githubUrl":"https://github.com/deepset-ai/haystack/blob/e318778c9bf60a1963e3b5f451359655dd696c30/haystack/evaluation/eval_run_result.py#L37-L73","documentation":"EvaluationRunResult requires each metric's outputs to include an 'individual_scores' list holding per-input scores. When that key is absent the constructor raises this ValueError. It exists so detailed and comparative reports always have per-example data available.","triggerScenarios":"Constructing EvaluationRunResult with a results dict like {'metric_name': {'score': 0.5}} where the per-input 'individual_scores' list was omitted.","commonSituations":"Custom metrics that only return an aggregate score; loading truncated evaluation output from JSON; hand-rolled evaluation pipelines that skip per-item scoring.","solutions":["Add an 'individual_scores' list to each metric's result dict with one score per input.","Make your metric callable return per-item scores, or compute them alongside the aggregate.","Pre-validate results: assert 'individual_scores' in metric_result for each metric."],"exampleFix":"// before\nresults = {'exact_match': {'score': 0.5}}\n// after\nresults = {'exact_match': {'score': 0.5, 'individual_scores': [1, 0]}}","handlingStrategy":"validation","validationCode":"for metric, outputs in results.items():\n    if 'individual_scores' not in outputs:\n        raise ValueError(f\"Metric '{metric}' has no 'individual_scores'\")","typeGuard":"def has_individual_scores(metric_output: dict) -> bool:\n    return isinstance(metric_output, dict) and isinstance(metric_output.get('individual_scores'), list)","tryCatchPattern":"try:\n    run = EvaluationRunResult(run_name, inputs, results)\nexcept ValueError as e:\n    if 'Individual scores missing' in str(e):\n        for m, o in results.items():\n            o.setdefault('individual_scores', [None] * len(next(iter(inputs.values()))))\n        run = EvaluationRunResult(run_name, inputs, results)\n    else:\n        raise","preventionTips":["Make every metric callable return per-item scores","Validate result dict keys after deserializing saved runs","Never hand-strip keys when post-processing results"],"tags":["python","validation","evaluation"],"backgroundTag":"missing-required-field","analyzedSha":"e318778c9bf60a1963e3b5f451359655dd696c30","analyzedAt":"2026-08-30T11:45:20.711Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}