run-llama/llama_index · error · ValueError

nested inner values in eval_kwargs must be a list. Got {eval

Error message

nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}

What it means

Raised by BatchRunner._validate_nested_eval_kwargs_types in the multi-evaluator form: eval_kwargs_lists is Dict[str, Dict[str, List]], and for some evaluator every inner value must itself be a list. This error names the exact evaluator, key, and offending value.

Source

Thrown at llama-index-core/llama_index/core/evaluation/batch_runner.py:171

        """
        if not isinstance(eval_kwargs_lists, dict):
            raise ValueError(
                f"eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}"
            )

        for evaluator, eval_kwargs in eval_kwargs_lists.items():
            if isinstance(eval_kwargs, list):
                # maintain backwards compatibility - for use with single evaluator
                eval_kwargs_lists[evaluator] = self._validate_and_clean_inputs(
                    eval_kwargs
                )[0]
            elif isinstance(eval_kwargs, dict):
                # for use with multiple evaluators
                for k in eval_kwargs:
                    v = eval_kwargs[k]
                    if not isinstance(v, list):
                        raise ValueError(
                            f"nested inner values in eval_kwargs must be a list. Got {evaluator}: {k}: {v}"
                        )
                    eval_kwargs_lists[evaluator][k] = self._validate_and_clean_inputs(
                        v
                    )[0]
            else:
                raise ValueError(
                    f"eval_kwargs must be a list or a dict. Got {evaluator}: {eval_kwargs}"
                )
        return eval_kwargs_lists

    def _get_eval_kwargs(
        self, eval_kwargs_lists: Dict[str, Any], idx: int
    ) -> Dict[str, Any]:
        """
        Get eval kwargs from eval_kwargs_lists at a given idx.

        Since eval_kwargs_lists is a dict of lists, we need to get the

View on GitHub (pinned to afd0fef371)

Solutions

1. Wrap inner values in a list of the same length as the queries: {"correctness": {"reference": [ref] * len(queries)}}.
2. If each query has its own reference, supply a per-item list: {"correctness": {"reference": [ref1, ref2, ...]}}.

Example fix

# before
await runner.aevaluate_responses(
    queries=queries, responses=responses,
    correctness={"reference": reference_answer},  # str -> raises
)

# after
await runner.aevaluate_responses(
    queries=queries, responses=responses,
    correctness={"reference": [reference_answer] * len(queries)},
)

def validate_nested(kw: dict, n: int) -> dict:
    for evaluator, inner in kw.items():
        if isinstance(inner, dict):
            for k, v in inner.items():
                if not isinstance(v, list):
                    inner[k] = [v] * n  # broadcast scalar
    return kw

def nested_values_are_lists(kw: dict) -> bool:
    return all(
        isinstance(v, list)
        for inner in kw.values() if isinstance(inner, dict)
        for v in inner.values()
    )

When it happens

Trigger: Calling aevaluate_responses with nested kwargs like {"correctness": {"reference": "some string"}} — the inner value is a str, not List[str]. Each inner list is then length-validated and index-sliced per item, so non-lists are rejected.

Common situations: Assuming a kwarg can be a scalar shared across all items; copy-pasting a single reference answer into the nested structure; migrating from the legacy flat-list form to the nested multi-evaluator form and wrapping values incorrectly.

Related errors

• eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}
• eval_kwargs must be a list or a dict. Got {evaluator}: {eval
• Unexpected type: {type(choice)}
• Unexpected type: {type(query)}
• embeddings_cache must be of type BaseKVStore

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/c17ba0c4b11623c6. Report an issue: GitHub.