{"record":{"id":"c17ba0c4b11623c6","repo":"run-llama/llama_index","slug":"nested-inner-values-in-eval-kwargs-must-be-a-list","errorCode":null,"errorMessage":"nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}","messagePattern":"nested inner values in eval_kwargs must be a list\\. Got (.+?): (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/batch_runner.py","lineNumber":171,"sourceCode":"\n        \"\"\"\n        if not isinstance(eval_kwargs_lists, dict):\n            raise ValueError(\n                f\"eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}\"\n            )\n\n        for evaluator, eval_kwargs in eval_kwargs_lists.items():\n            if isinstance(eval_kwargs, list):\n                # maintain backwards compatibility - for use with single evaluator\n                eval_kwargs_lists[evaluator] = self._validate_and_clean_inputs(\n                    eval_kwargs\n                )[0]\n            elif isinstance(eval_kwargs, dict):\n                # for use with multiple evaluators\n                for k in eval_kwargs:\n                    v = eval_kwargs[k]\n                    if not isinstance(v, list):\n                        raise ValueError(\n                            f\"nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}\"\n                        )\n                    eval_kwargs_lists[evaluator][k] = self._validate_and_clean_inputs(\n                        v\n                    )[0]\n            else:\n                raise ValueError(\n                    f\"eval_kwargs must be a list or a dict. Got {evaluator}: {eval_kwargs}\"\n                )\n        return eval_kwargs_lists\n\n    def _get_eval_kwargs(\n        self, eval_kwargs_lists: Dict[str, Any], idx: int\n    ) -> Dict[str, Any]:\n        \"\"\"\n        Get eval kwargs from eval_kwargs_lists at a given idx.\n\n        Since eval_kwargs_lists is a dict of lists, we need to get the","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/batch_runner.py#L153-L189","documentation":"Raised by BatchRunner._validate_nested_eval_kwargs_types in the multi-evaluator form: eval_kwargs_lists is Dict[str, Dict[str, List]], and for some evaluator every inner value must itself be a list. This error names the exact evaluator, key, and offending value.","triggerScenarios":"Calling aevaluate_responses with nested kwargs like {\"correctness\": {\"reference\": \"some string\"}} — the inner value is a str, not List[str]. Each inner list is then length-validated and index-sliced per item, so non-lists are rejected.","commonSituations":"Assuming a kwarg can be a scalar shared across all items; copy-pasting a single reference answer into the nested structure; migrating from the legacy flat-list form to the nested multi-evaluator form and wrapping values incorrectly.","solutions":["Wrap inner values in a list of the same length as the queries: {\"correctness\": {\"reference\": [ref] * len(queries)}}.","If each query has its own reference, supply a per-item list: {\"correctness\": {\"reference\": [ref1, ref2, ...]}}."],"exampleFix":"# before\nawait runner.aevaluate_responses(\n    queries=queries, responses=responses,\n    correctness={\"reference\": reference_answer},  # str -> raises\n)\n\n# after\nawait runner.aevaluate_responses(\n    queries=queries, responses=responses,\n    correctness={\"reference\": [reference_answer] * len(queries)},\n)","handlingStrategy":"type-guard","validationCode":"def validate_nested(kw: dict, n: int) -> dict:\n    for evaluator, inner in kw.items():\n        if isinstance(inner, dict):\n            for k, v in inner.items():\n                if not isinstance(v, list):\n                    inner[k] = [v] * n  # broadcast scalar\n    return kw","typeGuard":"def nested_values_are_lists(kw: dict) -> bool:\n    return all(\n        isinstance(v, list)\n        for inner in kw.values() if isinstance(inner, dict)\n        for v in inner.values()\n    )","tryCatchPattern":null,"preventionTips":["Remember the shape: top-level keys are evaluator names; inner values are per-item lists.","Broadcast scalars explicitly: [value] * len(queries)."],"tags":["batch-evaluation","type-validation","nested-kwargs"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}