What causes "The response is invalid" in run-llama/llama_index?

Pass raise_error=False (default) to get passing=False and score=0.0 instead of an exception

run-llama/llama_index · error · ValueError

The response is invalid

Error message

The response is invalid

What it means

Raised by MultiModalRelevancyEvaluator.evaluate only when raise_error=True and the judge LLM's answer does not contain 'yes' (case-insensitive). The evaluator asks the LLM whether contexts/response are relevant and scans for the substring 'yes'; a negative or unparsable verdict normally yields passing=False, but with raise_error=True it converts the failure into a ValueError.

Source

Thrown at llama-index-core/llama_index/core/evaluation/multi_modal/relevancy.py:154

            blocks.extend(
                [ImageBlock(path=Path(image_path)) for image_path in image_paths]
            )
        if image_urls:
            blocks.extend([ImageBlock(url=image_url) for image_url in image_urls])

        blocks.append(TextBlock(text=fmt_prompt))

        response_obj = self._multi_modal_llm.chat(
            messages=[ChatMessage(role="user", blocks=blocks)],
        )

        raw_response_txt: str = response_obj.message.content or ""

        if "yes" in raw_response_txt.lower():
            passing = True
        else:
            if self._raise_error:
                raise ValueError("The response is invalid")
            passing = False

        return EvaluationResult(
            query=query,
            response=response,
            passing=passing,
            score=1.0 if passing else 0.0,
            feedback=raw_response_txt,
        )

    async def aevaluate(
        self,
        query: Union[str, None] = None,
        response: Union[str, None] = None,
        contexts: Union[Sequence[str], None] = None,
        image_paths: Union[List[str], None] = None,
        image_urls: Union[List[str], None] = None,
        **kwargs: Any,

View on GitHub (pinned to afd0fef371)

Solutions

Pass raise_error=False (default) to get passing=False and score=0.0 instead of an exception
If the verdict seems wrong, inspect result.feedback (raw judge text) with raise_error=False to debug the judge's answer
Check that your contexts/response are actually relevant and that the eval template is being formatted correctly

Example fix

# before
evaluator = MultiModalRelevancyEvaluator(raise_error=True)
result = evaluator.evaluate(query=q, contexts=ctxs, response=r)  # may raise

# after
evaluator = MultiModalRelevancyEvaluator(raise_error=False)
result = evaluator.evaluate(query=q, contexts=ctxs, response=r)
assert result.passing or result.feedback  # inspect judge verdict

Defensive patterns

Strategy: try-catch

Try / catch

try:
    result = evaluator.evaluate(query=q, contexts=ctxs, response=resp)
except ValueError as e:
    if str(e) == "The response is invalid":
        result = None  # treat as failed eval, inspect separately
    else:
        raise

Prevention

Prefer raise_error=False and check result.passing instead of exceptions
Keep temperature 0 and a deterministic judge to reduce flaky 'no' verdicts
Persist result.feedback for every failing verdict to audit judge behavior

When it happens

Trigger: Constructing the evaluator with raise_error=True and evaluating a query/context pair the judge deems irrelevant; judge returns 'NO', 'no.', or any text without the substring 'yes' (including malformed/refusal responses).

Common situations: Strict eval pipelines that want hard failures on irrelevant retrievals; low-temperature judges that answer 'No'; models returning empty content or refusals; substring false-negatives from unusual phrasing.

Related errors

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/c34e05ed7a478c5f. Report an issue: GitHub.