run-llama/llama_index · error · ValueError

query, contexts, and response must be provided

Error message

query, contexts, and response must be provided

What it means

Raised by MultiModalRelevancyEvaluator.evaluate when any of query, contexts, or response is None. These three parameters are required to build the evaluation prompt (context_str and evaluation_query_str); the signature marks them Optional only to allow keyword omission, and the guard rejects None before any LLM call.

Source

Thrown at llama-index-core/llama_index/core/evaluation/multi_modal/relevancy.py:125

        if "eval_template" in prompts:
            self._eval_template = prompts["eval_template"]
        if "refine_template" in prompts:
            self._refine_template = prompts["refine_template"]

    def evaluate(
        self,
        query: Union[str, None] = None,
        response: Union[str, None] = None,
        contexts: Union[Sequence[str], None] = None,
        image_paths: Union[List[str], None] = None,
        image_urls: Union[List[str], None] = None,
        **kwargs: Any,
    ) -> EvaluationResult:
        """Evaluate whether the multi-modal contexts and response are relevant to the query."""
        del kwargs  # Unused

        if query is None or contexts is None or response is None:
            raise ValueError("query, contexts, and response must be provided")

        context_str = "\n\n".join(contexts)
        evaluation_query_str = f"Question: {query}\nResponse: {response}"
        fmt_prompt = self._eval_template.format(
            context_str=context_str, query_str=evaluation_query_str
        )

        blocks: List[Union[ImageBlock, TextBlock]] = []

        if image_paths:
            blocks.extend(
                [ImageBlock(path=Path(image_path)) for image_path in image_paths]
            )
        if image_urls:
            blocks.extend([ImageBlock(url=image_url) for image_url in image_urls])

        blocks.append(TextBlock(text=fmt_prompt))

View on GitHub (pinned to afd0fef371)

Solutions

1. Ensure all three values are non-None strings/lists before calling: evaluate(query=q, contexts=ctxs, response=resp)
2. Filter or skip samples with missing fields in your eval loop before invoking the evaluator
3. Default missing values explicitly, e.g. response='' or contexts=[] if you intentionally want empty inputs evaluated

Example fix

# before
result = evaluator.evaluate(query=q, contexts=None, response=r)

# after
if q is None or r is None or ctxs is None:
    continue  # skip incomplete sample
result = evaluator.evaluate(query=q, contexts=ctxs, response=r)

def can_evaluate(query, contexts, response) -> bool:
    return query is not None and contexts is not None and response is not None

if can_evaluate(q, ctxs, resp):
    result = evaluator.evaluate(query=q, contexts=ctxs, response=resp)

try:
    result = evaluator.evaluate(query=q, contexts=ctxs, response=resp)
except ValueError as e:
    if "must be provided" in str(e):
        logger.warning("skipping incomplete sample")
        continue
    raise

When it happens

Trigger: Calling evaluate(query=..., contexts=None, response=...) or omitting any of the three kwargs; passing an empty pipeline result (e.g. response=None because the RAG step failed or was skipped).

Common situations: Wiring an eval loop where some samples lack retrieved contexts or a generated response; mapping over batches where a field is None for failed items; renaming kwargs (answer vs response).

Related errors

• names and results_arr must have same length.
• query, response, second_response, and reference must be prov
• query, contexts, and response must be provided
• Retrieved ids and expected ids must be provided
• query and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/70992b81cdfd3e97. Report an issue: GitHub.