run-llama/llama_index · error · ValueError

The response is invalid

Error message

The response is invalid

What it means

Thrown by MultiModalFaithfulnessEvaluator.evaluate (sync) when raise_error=True and the multi-modal LLM's chat reply does not contain 'yes' (case-insensitive) or is empty. The evaluator checks response_obj.message.content for the substring and treats everything else — including genuine 'No' verdicts and empty content — as invalid.

Source

Thrown at llama-index-core/llama_index/core/evaluation/multi_modal/faithfulness.py:175

                [ImageBlock(path=Path(image_path)) for image_path in image_paths]
            )
        if image_urls:
            image_nodes.extend([ImageBlock(url=image_url) for image_url in image_urls])

        image_nodes.append(TextBlock(text=fmt_prompt))

        response_obj = self._multi_modal_llm.chat(
            messages=[ChatMessage(role="user", blocks=image_nodes)],
        )

        raw_response_txt: str = response_obj.message.content or ""

        if "yes" in raw_response_txt.lower():
            passing = True
        else:
            passing = False
            if self._raise_error:
                raise ValueError("The response is invalid")

        return EvaluationResult(
            response=response,
            contexts=contexts,
            passing=passing,
            score=1.0 if passing else 0.0,
            feedback=raw_response_txt,
        )

    async def aevaluate(
        self,
        query: Union[str, None] = None,
        response: Union[str, None] = None,
        contexts: Union[Sequence[str], None] = None,
        image_paths: Union[List[str], None] = None,
        image_urls: Union[List[str], None] = None,
        **kwargs: Any,
    ) -> EvaluationResult:

View on GitHub (pinned to afd0fef371)

Solutions

1. Use raise_error=False and branch on result.passing / inspect result.feedback to distinguish real 'No' verdicts from empty outputs.
2. Increase max_output_tokens on the multi-modal LLM (the default fallback uses 1000) and verify images are readable paths/URLs.
3. Log raw feedback for every non-passing item so empty-content failures surface quickly.

Example fix

# before
evaluator = MultiModalFaithfulnessEvaluator(raise_error=True)
result = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)

# after
evaluator = MultiModalFaithfulnessEvaluator(raise_error=False)
result = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)
if not result.passing:
    logger.warning("verdict feedback: %r", result.feedback)

result = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)
if not result.passing:
    logger.warning("mm faithfulness feedback: %r", result.feedback)

def mm_faithful(r) -> bool:
    return bool(r.passing)

try:
    result = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)
except ValueError as e:
    if e.args[0] == "The response is invalid":
        return None
    raise

When it happens

Trigger: evaluator.evaluate(response=..., contexts=..., image_paths=[...]) with raise_error=True where the model replies 'No.', gives an explanation, or returns empty content (None coerced to "").

Common situations: Models returning empty content due to safety filters or output-token limits; verbose models that explain instead of answering YES/NO; strict pipelines with raise_error=True aborting on any negative verdict; images the model cannot read returning placeholder text.

Related errors

• The response is invalid
• contexts and response must be provided
• The response is invalid
• Unknown tool name: {tool_call_result.tool_name}
• Unhandled shape {array.shape}.

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/223daba18faf8adf. Report an issue: GitHub.