{"record":{"id":"3d60b4c2376b6e06","repo":"run-llama/llama_index","slug":"eval-kwargs-must-be-a-list-or-a-dict-got-evaluat","errorCode":null,"errorMessage":"eval_kwargs must be a list or a dict. Got {evaluator}: {eval_kwargs}","messagePattern":"eval_kwargs must be a list or a dict\\. Got (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/batch_runner.py","lineNumber":178,"sourceCode":"        for evaluator, eval_kwargs in eval_kwargs_lists.items():\n            if isinstance(eval_kwargs, list):\n                # maintain backwards compatibility - for use with single evaluator\n                eval_kwargs_lists[evaluator] = self._validate_and_clean_inputs(\n                    eval_kwargs\n                )[0]\n            elif isinstance(eval_kwargs, dict):\n                # for use with multiple evaluators\n                for k in eval_kwargs:\n                    v = eval_kwargs[k]\n                    if not isinstance(v, list):\n                        raise ValueError(\n                            f\"nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}\"\n                        )\n                    eval_kwargs_lists[evaluator][k] = self._validate_and_clean_inputs(\n                        v\n                    )[0]\n            else:\n                raise ValueError(\n                    f\"eval_kwargs must be a list or a dict. Got {evaluator}: {eval_kwargs}\"\n                )\n        return eval_kwargs_lists\n\n    def _get_eval_kwargs(\n        self, eval_kwargs_lists: Dict[str, Any], idx: int\n    ) -> Dict[str, Any]:\n        \"\"\"\n        Get eval kwargs from eval_kwargs_lists at a given idx.\n\n        Since eval_kwargs_lists is a dict of lists, we need to get the\n        value at idx for each key.\n\n        \"\"\"\n        return {k: v[idx] for k, v in eval_kwargs_lists.items()}\n\n    async def aevaluate_response_strs(\n        self,","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/batch_runner.py#L160-L196","documentation":"Raised by BatchRunner._validate_nested_eval_kwargs_types when a per-evaluator value in eval_kwargs_lists is neither a list (legacy single-evaluator form) nor a dict (multi-evaluator form). The message names the evaluator key and the offending value.","triggerScenarios":"Passing kwargs like correctness=\"some string\", correctness=42, or correctness=(\"a\", \"b\") — any non-list, non-dict value under an evaluator key.","commonSituations":"Setting a scalar config value (e.g. a threshold string) as an eval kwarg, which the runner interprets as an evaluator name; tuple instead of list; misunderstanding that top-level kwargs keys are evaluator names and values are per-item lists.","solutions":["Use the correct shape: correctness=[...] for one evaluator or correctness={\"reference\": [...]} for per-evaluator kwargs.","Remove config-style kwargs that are not per-item lists — they do not belong in **eval_kwargs_lists."],"exampleFix":"# before\nawait runner.aevaluate_responses(\n    queries=queries, responses=responses, correctness=\"ref answer\"\n)\n\n# after\nawait runner.aevaluate_responses(\n    queries=queries, responses=responses,\n    correctness={\"reference\": [\"ref answer\"] * len(queries)},\n)","handlingStrategy":"type-guard","validationCode":"for evaluator, v in eval_kwargs_lists.items():\n    if not isinstance(v, (list, dict)):\n        raise TypeError(f\"bad eval kwargs for {evaluator!r}: expected list or dict\")","typeGuard":"def is_eval_kwargs_shape(v) -> bool:\n    return isinstance(v, (list, dict))","tryCatchPattern":null,"preventionTips":["Keep config-style settings out of **eval_kwargs_lists — it only carries per-item lists.","Convert tuples to lists before passing."],"tags":["batch-evaluation","type-validation","api-misuse"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}