{"record":{"id":"c34e05ed7a478c5f","repo":"run-llama/llama_index","slug":"the-response-is-invalid-c34e05","errorCode":null,"errorMessage":"The response is invalid","messagePattern":"The response is invalid","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/multi_modal/relevancy.py","lineNumber":154,"sourceCode":"            blocks.extend(\n                [ImageBlock(path=Path(image_path)) for image_path in image_paths]\n            )\n        if image_urls:\n            blocks.extend([ImageBlock(url=image_url) for image_url in image_urls])\n\n        blocks.append(TextBlock(text=fmt_prompt))\n\n        response_obj = self._multi_modal_llm.chat(\n            messages=[ChatMessage(role=\"user\", blocks=blocks)],\n        )\n\n        raw_response_txt: str = response_obj.message.content or \"\"\n\n        if \"yes\" in raw_response_txt.lower():\n            passing = True\n        else:\n            if self._raise_error:\n                raise ValueError(\"The response is invalid\")\n            passing = False\n\n        return EvaluationResult(\n            query=query,\n            response=response,\n            passing=passing,\n            score=1.0 if passing else 0.0,\n            feedback=raw_response_txt,\n        )\n\n    async def aevaluate(\n        self,\n        query: Union[str, None] = None,\n        response: Union[str, None] = None,\n        contexts: Union[Sequence[str], None] = None,\n        image_paths: Union[List[str], None] = None,\n        image_urls: Union[List[str], None] = None,\n        **kwargs: Any,","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/multi_modal/relevancy.py#L136-L172","documentation":"Raised by MultiModalRelevancyEvaluator.evaluate only when raise_error=True and the judge LLM's answer does not contain 'yes' (case-insensitive). The evaluator asks the LLM whether contexts/response are relevant and scans for the substring 'yes'; a negative or unparsable verdict normally yields passing=False, but with raise_error=True it converts the failure into a ValueError.","triggerScenarios":"Constructing the evaluator with raise_error=True and evaluating a query/context pair the judge deems irrelevant; judge returns 'NO', 'no.', or any text without the substring 'yes' (including malformed/refusal responses).","commonSituations":"Strict eval pipelines that want hard failures on irrelevant retrievals; low-temperature judges that answer 'No'; models returning empty content or refusals; substring false-negatives from unusual phrasing.","solutions":["Pass raise_error=False (default) to get passing=False and score=0.0 instead of an exception","If the verdict seems wrong, inspect result.feedback (raw judge text) with raise_error=False to debug the judge's answer","Check that your contexts/response are actually relevant and that the eval template is being formatted correctly"],"exampleFix":"# before\nevaluator = MultiModalRelevancyEvaluator(raise_error=True)\nresult = evaluator.evaluate(query=q, contexts=ctxs, response=r)  # may raise\n\n# after\nevaluator = MultiModalRelevancyEvaluator(raise_error=False)\nresult = evaluator.evaluate(query=q, contexts=ctxs, response=r)\nassert result.passing or result.feedback  # inspect judge verdict","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    result = evaluator.evaluate(query=q, contexts=ctxs, response=resp)\nexcept ValueError as e:\n    if str(e) == \"The response is invalid\":\n        result = None  # treat as failed eval, inspect separately\n    else:\n        raise","preventionTips":["Prefer raise_error=False and check result.passing instead of exceptions","Keep temperature 0 and a deterministic judge to reduce flaky 'no' verdicts","Persist result.feedback for every failing verdict to audit judge behavior"],"tags":["evaluation","llm-verdict","multi-modal","strict-mode"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}