{"record":{"id":"223daba18faf8adf","repo":"run-llama/llama_index","slug":"the-response-is-invalid-223dab","errorCode":null,"errorMessage":"The response is invalid","messagePattern":"The response is invalid","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/multi_modal/faithfulness.py","lineNumber":175,"sourceCode":"                [ImageBlock(path=Path(image_path)) for image_path in image_paths]\n            )\n        if image_urls:\n            image_nodes.extend([ImageBlock(url=image_url) for image_url in image_urls])\n\n        image_nodes.append(TextBlock(text=fmt_prompt))\n\n        response_obj = self._multi_modal_llm.chat(\n            messages=[ChatMessage(role=\"user\", blocks=image_nodes)],\n        )\n\n        raw_response_txt: str = response_obj.message.content or \"\"\n\n        if \"yes\" in raw_response_txt.lower():\n            passing = True\n        else:\n            passing = False\n            if self._raise_error:\n                raise ValueError(\"The response is invalid\")\n\n        return EvaluationResult(\n            response=response,\n            contexts=contexts,\n            passing=passing,\n            score=1.0 if passing else 0.0,\n            feedback=raw_response_txt,\n        )\n\n    async def aevaluate(\n        self,\n        query: Union[str, None] = None,\n        response: Union[str, None] = None,\n        contexts: Union[Sequence[str], None] = None,\n        image_paths: Union[List[str], None] = None,\n        image_urls: Union[List[str], None] = None,\n        **kwargs: Any,\n    ) -> EvaluationResult:","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/multi_modal/faithfulness.py#L157-L193","documentation":"Thrown by MultiModalFaithfulnessEvaluator.evaluate (sync) when raise_error=True and the multi-modal LLM's chat reply does not contain 'yes' (case-insensitive) or is empty. The evaluator checks response_obj.message.content for the substring and treats everything else — including genuine 'No' verdicts and empty content — as invalid.","triggerScenarios":"evaluator.evaluate(response=..., contexts=..., image_paths=[...]) with raise_error=True where the model replies 'No.', gives an explanation, or returns empty content (None coerced to \"\").","commonSituations":"Models returning empty content due to safety filters or output-token limits; verbose models that explain instead of answering YES/NO; strict pipelines with raise_error=True aborting on any negative verdict; images the model cannot read returning placeholder text.","solutions":["Use raise_error=False and branch on result.passing / inspect result.feedback to distinguish real 'No' verdicts from empty outputs.","Increase max_output_tokens on the multi-modal LLM (the default fallback uses 1000) and verify images are readable paths/URLs.","Log raw feedback for every non-passing item so empty-content failures surface quickly."],"exampleFix":"# before\nevaluator = MultiModalFaithfulnessEvaluator(raise_error=True)\nresult = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)\n\n# after\nevaluator = MultiModalFaithfulnessEvaluator(raise_error=False)\nresult = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)\nif not result.passing:\n    logger.warning(\"verdict feedback: %r\", result.feedback)","handlingStrategy":"fallback","validationCode":"result = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)\nif not result.passing:\n    logger.warning(\"mm faithfulness feedback: %r\", result.feedback)","typeGuard":"def mm_faithful(r) -> bool:\n    return bool(r.passing)","tryCatchPattern":"try:\n    result = evaluator.evaluate(response=r, contexts=ctxs, image_paths=paths)\nexcept ValueError as e:\n    if e.args[0] == \"The response is invalid\":\n        return None\n    raise","preventionTips":["Prefer raise_error=False and inspect feedback; empty model content is common with multimodal calls.","Verify image paths/URLs are accessible and max_output_tokens is sufficient before strict runs."],"tags":["multi-modal","llm-output","faithfulness","strict-mode"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}