run-llama/llama_index · error · ValueError

eval_kwargs must be a list or a dict. Got {evaluator}: {eval

Error message

eval_kwargs must be a list or a dict. Got {evaluator}: {eval_kwargs}

What it means

Raised by BatchRunner._validate_nested_eval_kwargs_types when a per-evaluator value in eval_kwargs_lists is neither a list (legacy single-evaluator form) nor a dict (multi-evaluator form). The message names the evaluator key and the offending value.

Source

Thrown at llama-index-core/llama_index/core/evaluation/batch_runner.py:178

        for evaluator, eval_kwargs in eval_kwargs_lists.items():
            if isinstance(eval_kwargs, list):
                # maintain backwards compatibility - for use with single evaluator
                eval_kwargs_lists[evaluator] = self._validate_and_clean_inputs(
                    eval_kwargs
                )[0]
            elif isinstance(eval_kwargs, dict):
                # for use with multiple evaluators
                for k in eval_kwargs:
                    v = eval_kwargs[k]
                    if not isinstance(v, list):
                        raise ValueError(
                            f"nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}"
                        )
                    eval_kwargs_lists[evaluator][k] = self._validate_and_clean_inputs(
                        v
                    )[0]
            else:
                raise ValueError(
                    f"eval_kwargs must be a list or a dict. Got {evaluator}: {eval_kwargs}"
                )
        return eval_kwargs_lists

    def _get_eval_kwargs(
        self, eval_kwargs_lists: Dict[str, Any], idx: int
    ) -> Dict[str, Any]:
        """
        Get eval kwargs from eval_kwargs_lists at a given idx.

        Since eval_kwargs_lists is a dict of lists, we need to get the
        value at idx for each key.

        """
        return {k: v[idx] for k, v in eval_kwargs_lists.items()}

    async def aevaluate_response_strs(
        self,

View on GitHub (pinned to afd0fef371)

Solutions

1. Use the correct shape: correctness=[...] for one evaluator or correctness={"reference": [...]} for per-evaluator kwargs.
2. Remove config-style kwargs that are not per-item lists — they do not belong in **eval_kwargs_lists.

Example fix

# before
await runner.aevaluate_responses(
    queries=queries, responses=responses, correctness="ref answer"
)

# after
await runner.aevaluate_responses(
    queries=queries, responses=responses,
    correctness={"reference": ["ref answer"] * len(queries)},
)

for evaluator, v in eval_kwargs_lists.items():
    if not isinstance(v, (list, dict)):
        raise TypeError(f"bad eval kwargs for {evaluator!r}: expected list or dict")

def is_eval_kwargs_shape(v) -> bool:
    return isinstance(v, (list, dict))

When it happens

Trigger: Passing kwargs like correctness="some string", correctness=42, or correctness=("a", "b") — any non-list, non-dict value under an evaluator key.

Common situations: Setting a scalar config value (e.g. a threshold string) as an eval kwarg, which the runner interprets as an evaluator name; tuple instead of list; misunderstanding that top-level kwargs keys are evaluator names and values are per-item lists.

Related errors

• eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}
• nested inner values in eval_kwargs must be a list. Got {eval
• `queries` must be provided
• This query engine does not support retrieve, use query direc
• This query engine does not support synthesize, use query dir

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/3d60b4c2376b6e06. Report an issue: GitHub.