{"record":{"id":"c3b5b0f0dad54ef4","repo":"run-llama/llama_index","slug":"eval-kwargs-lists-must-be-a-dict-got-eval-kwargs","errorCode":null,"errorMessage":"eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}","messagePattern":"eval_kwargs_lists must be a dict\\. Got (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/batch_runner.py","lineNumber":156,"sourceCode":"            else:\n                if len(inputs) != input_len:\n                    raise ValueError(\"All inputs must have the same length.\")\n                new_inputs_list.append(inputs)\n        return new_inputs_list\n\n    def _validate_nested_eval_kwargs_types(\n        self, eval_kwargs_lists: Dict[str, Any]\n    ) -> Dict[str, Any]:\n        \"\"\"\n        Ensure eval kwargs are acceptable format.\n            either a Dict[str, List] or a Dict[str, Dict[str, List]].\n\n        Allows use of different kwargs (e.g. references) with different evaluators\n            while keeping backwards compatibility for single evaluators\n\n        \"\"\"\n        if not isinstance(eval_kwargs_lists, dict):\n            raise ValueError(\n                f\"eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}\"\n            )\n\n        for evaluator, eval_kwargs in eval_kwargs_lists.items():\n            if isinstance(eval_kwargs, list):\n                # maintain backwards compatibility - for use with single evaluator\n                eval_kwargs_lists[evaluator] = self._validate_and_clean_inputs(\n                    eval_kwargs\n                )[0]\n            elif isinstance(eval_kwargs, dict):\n                # for use with multiple evaluators\n                for k in eval_kwargs:\n                    v = eval_kwargs[k]\n                    if not isinstance(v, list):\n                        raise ValueError(\n                            f\"nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}\"\n                        )\n                    eval_kwargs_lists[evaluator][k] = self._validate_and_clean_inputs(","sourceCodeStart":138,"sourceCodeEnd":174,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/batch_runner.py#L138-L174","documentation":"Raised by BatchRunner._validate_nested_eval_kwargs_types when the eval_kwargs_lists argument passed to aevaluate/aevaluate_responses (collected via **kwargs) is not a Python dict. The nested-kwargs API expects Dict[str, List] (single evaluator, legacy) or Dict[str, Dict[str, List]] (multiple evaluators).","triggerScenarios":"Calling await runner.aevaluate_queries(query_engine=..., queries=..., some_eval_kwargs) where some_eval_kwargs is a list, tuple, string, or None instead of a dict; or programmatically passing an unpacked variable that is not a dict.","commonSituations":"Confusing the kwargs API shape (passing a bare list of references instead of references=[...] keyword form); a helper function that conditionally sets eval_kwargs to None; passing a JSON string that was never parsed.","solutions":["Pass eval kwargs as keyword arguments so Python packs them into a dict: references=[...], or as a dict when calling the internal API directly.","If building kwargs dynamically, ensure the variable is a dict ({} when empty) before **-unpacking or passing."],"exampleFix":"# before\nawait runner.aevaluate(query_engine, queries, [ref1, ref2])  # bare list -> raises\n\n# after\nawait runner.aevaluate(query_engine, queries, references=[ref1, ref2])","handlingStrategy":"type-guard","validationCode":"if not isinstance(eval_kwargs_lists, dict):\n    eval_kwargs_lists = {}  # or raise with context before calling the runner","typeGuard":"def is_valid_kwargs_dict(kw) -> bool:\n    return isinstance(kw, dict) and all(\n        isinstance(v, (list, dict)) for v in kw.values()\n    )","tryCatchPattern":null,"preventionTips":["Pass eval kwargs as keyword arguments (references=[...]) instead of positionally-built structures.","Default dynamically-built kwargs to {} not None."],"tags":["batch-evaluation","type-validation","api-misuse"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}