run-llama/llama_index · error · ValueError

eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}

Error message

eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}

What it means

Raised by BatchRunner._validate_nested_eval_kwargs_types when the eval_kwargs_lists argument passed to aevaluate/aevaluate_responses (collected via **kwargs) is not a Python dict. The nested-kwargs API expects Dict[str, List] (single evaluator, legacy) or Dict[str, Dict[str, List]] (multiple evaluators).

Source

Thrown at llama-index-core/llama_index/core/evaluation/batch_runner.py:156

            else:
                if len(inputs) != input_len:
                    raise ValueError("All inputs must have the same length.")
                new_inputs_list.append(inputs)
        return new_inputs_list

    def _validate_nested_eval_kwargs_types(
        self, eval_kwargs_lists: Dict[str, Any]
    ) -> Dict[str, Any]:
        """
        Ensure eval kwargs are acceptable format.
            either a Dict[str, List] or a Dict[str, Dict[str, List]].

        Allows use of different kwargs (e.g. references) with different evaluators
            while keeping backwards compatibility for single evaluators

        """
        if not isinstance(eval_kwargs_lists, dict):
            raise ValueError(
                f"eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}"
            )

        for evaluator, eval_kwargs in eval_kwargs_lists.items():
            if isinstance(eval_kwargs, list):
                # maintain backwards compatibility - for use with single evaluator
                eval_kwargs_lists[evaluator] = self._validate_and_clean_inputs(
                    eval_kwargs
                )[0]
            elif isinstance(eval_kwargs, dict):
                # for use with multiple evaluators
                for k in eval_kwargs:
                    v = eval_kwargs[k]
                    if not isinstance(v, list):
                        raise ValueError(
                            f"nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}"
                        )
                    eval_kwargs_lists[evaluator][k] = self._validate_and_clean_inputs(

View on GitHub (pinned to afd0fef371)

Solutions

1. Pass eval kwargs as keyword arguments so Python packs them into a dict: references=[...], or as a dict when calling the internal API directly.
2. If building kwargs dynamically, ensure the variable is a dict ({} when empty) before **-unpacking or passing.

Example fix

# before
await runner.aevaluate(query_engine, queries, [ref1, ref2])  # bare list -> raises

# after
await runner.aevaluate(query_engine, queries, references=[ref1, ref2])

if not isinstance(eval_kwargs_lists, dict):
    eval_kwargs_lists = {}  # or raise with context before calling the runner

def is_valid_kwargs_dict(kw) -> bool:
    return isinstance(kw, dict) and all(
        isinstance(v, (list, dict)) for v in kw.values()
    )

When it happens

Trigger: Calling await runner.aevaluate_queries(query_engine=..., queries=..., some_eval_kwargs) where some_eval_kwargs is a list, tuple, string, or None instead of a dict; or programmatically passing an unpacked variable that is not a dict.

Common situations: Confusing the kwargs API shape (passing a bare list of references instead of references=[...] keyword form); a helper function that conditionally sets eval_kwargs to None; passing a JSON string that was never parsed.

Related errors

• eval_kwargs must be a list or a dict. Got {evaluator}: {eval
• nested inner values in eval_kwargs must be a list. Got {eval
• `queries` must be provided
• This query engine does not support retrieve, use query direc
• This query engine does not support synthesize, use query dir

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/c3b5b0f0dad54ef4. Report an issue: GitHub.