run-llama/llama_index · error · ValueError

`queries` must be provided

Error message

`queries` must be provided

What it means

Raised by BatchRunner.aevaluate_queries when the queries argument is None. The runner needs an explicit list of query strings to dispatch through the query_engine via response_worker coroutines before evaluation.

Source

Thrown at llama-index-core/llama_index/core/evaluation/batch_runner.py:336

    async def aevaluate_queries(
        self,
        query_engine: BaseQueryEngine,
        queries: Optional[List[str]] = None,
        **eval_kwargs_lists: Dict[str, Any],
    ) -> Dict[str, List[EvaluationResult]]:
        """
        Evaluate queries.

        Args:
            query_engine (BaseQueryEngine): Query engine.
            queries (Optional[List[str]]): List of query strings. Defaults to None.
            **eval_kwargs_lists (Dict[str, Any]): Dict of lists of kwargs to
                pass to evaluator. Defaults to None.

        """
        if queries is None:
            raise ValueError("`queries` must be provided")

        # gather responses
        response_jobs = []
        for query in queries:
            response_jobs.append(response_worker(self.semaphore, query_engine, query))
        responses = await self.asyncio_mod.gather(*response_jobs)

        return await self.aevaluate_responses(
            queries=queries,
            responses=responses,
            **eval_kwargs_lists,
        )

    def evaluate_response_strs(
        self,
        queries: Optional[List[str]] = None,
        response_strs: Optional[List[str]] = None,
        contexts_list: Optional[List[List[str]]] = None,

View on GitHub (pinned to afd0fef371)

Solutions

1. Pass an explicit non-empty list of query strings: queries=["What is X?", ...].
2. Add a guard before the call: if not queries: raise/log early with context about why the list is empty.
3. Verify the upstream loader (dataset read, QueryDataset.queries) actually returned items.

Example fix

# before
queries = load_queries()  # returned None
await runner.aevaluate_queries(engine, queries)

# after
queries = load_queries() or []
if not queries:
    raise RuntimeError("no queries loaded from dataset")
await runner.aevaluate_queries(engine, queries)

if not queries:
    raise RuntimeError(f"refusing to evaluate: {len(queries) if queries else 0} queries")

def has_queries(queries) -> bool:
    return isinstance(queries, list) and len(queries) > 0 and all(isinstance(q, str) for q in queries)

When it happens

Trigger: Calling await runner.aevaluate_queries(query_engine=engine, queries=None); typically a variable that was supposed to hold a list of questions but was never populated or was conditionally set to None.

Common situations: Loading queries from a file/dataset that failed silently and left the variable None; branching code where one path forgets to assign queries; refactoring from aevaluate_responses to aevaluate_queries and dropping the argument.

Related errors

• At least one item in inputs_list must be provided.
• All inputs must have the same length.
• eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}
• eval_kwargs must be a list or a dict. Got {evaluator}: {eval
• code_execute_fn must be provided for CodeActAgent

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/669ab27d5f110ae9. Report an issue: GitHub.