run-llama/llama_index · error · ValueError

All inputs must have the same length.

Error message

All inputs must have the same length.

What it means

Raised by BatchRunner._validate_and_clean_inputs when two or more provided kwargs lists have different lengths. After the first non-None list fixes input_len, every other non-None list must match that length, because kwargs are zipped index-by-index with the queries/responses.

Source

Thrown at llama-index-core/llama_index/core/evaluation/batch_runner.py:140

        """
        assert len(inputs_list) > 0
        # first, make sure at least one of queries or response_strs is not None
        input_len: Optional[int] = None
        for inputs in inputs_list:
            if inputs is not None:
                input_len = len(inputs)
                break
        if input_len is None:
            raise ValueError("At least one item in inputs_list must be provided.")

        new_inputs_list = []
        for inputs in inputs_list:
            if inputs is None:
                new_inputs_list.append([None] * input_len)
            else:
                if len(inputs) != input_len:
                    raise ValueError("All inputs must have the same length.")
                new_inputs_list.append(inputs)
        return new_inputs_list

    def _validate_nested_eval_kwargs_types(
        self, eval_kwargs_lists: Dict[str, Any]
    ) -> Dict[str, Any]:
        """
        Ensure eval kwargs are acceptable format.
            either a Dict[str, List] or a Dict[str, Dict[str, List]].

        Allows use of different kwargs (e.g. references) with different evaluators
            while keeping backwards compatibility for single evaluators

        """
        if not isinstance(eval_kwargs_lists, dict):
            raise ValueError(
                f"eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}"
            )

View on GitHub (pinned to afd0fef371)

Solutions

1. Log len() of every list you pass and make them equal before calling the runner.
2. Align lists at construction time: build (query, response, reference) tuples first, then unzip, so lengths cannot diverge.
3. If some items lack a reference, pad with [None] * len(queries) at the right positions instead of shortening the list.

Example fix

# before
await runner.aevaluate_responses(
    queries=queries,               # len 10
    responses=responses,           # len 8  -> raises
)

# after
n = min(len(queries), len(responses))
await runner.aevaluate_responses(
    queries=queries[:n], responses=responses[:n]
)

def assert_aligned(queries, responses, **kwargs_lists):
    n = len(queries)
    for name, lst in {"responses": responses, **kwargs_lists}.items():
        if lst is not None and len(lst) != n:
            raise ValueError(f"{name} has {len(lst)} items, expected {n}")
    return True

def same_len(*lists) -> bool:
    lens = {len(x) for x in lists if x is not None}
    return len(lens) <= 1

When it happens

Trigger: Calling aevaluate_responses (directly or via aevaluate) with e.g. len(queries)=10 but len(responses)=8, or a references list whose length differs from the queries list.

Common situations: Building queries and references from different sources (queries from a dataset, references hand-written); filtering one list (dropping failed generations) without filtering the other; off-by-one when slicing lists.

Related errors

• At least one item in inputs_list must be provided.
• `queries` must be provided
• code_execute_fn must be provided for CodeActAgent
• eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}
• nested inner values in eval_kwargs must be a list. Got {eval

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/c51d4c115ed95d37. Report an issue: GitHub.