{"record":{"id":"dd1190cd6c4e1204","repo":"run-llama/llama_index","slug":"the-response-is-invalid-dd1190","errorCode":null,"errorMessage":"The response is invalid","messagePattern":"The response is invalid","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/relevancy.py","lineNumber":131,"sourceCode":"\n        query_response = f\"Question: {query}\\nResponse: {response}\"\n\n        await asyncio.sleep(sleep_time_in_seconds)\n\n        query_engine = index.as_query_engine(\n            llm=self._llm,\n            text_qa_template=self._eval_template,\n            refine_template=self._refine_template,\n        )\n        response_obj = await query_engine.aquery(query_response)\n\n        raw_response_txt = str(response_obj)\n\n        if \"yes\" in raw_response_txt.lower():\n            passing = True\n        else:\n            if self._raise_error:\n                raise ValueError(\"The response is invalid\")\n            passing = False\n\n        return EvaluationResult(\n            query=query,\n            response=response,\n            passing=passing,\n            score=1.0 if passing else 0.0,\n            feedback=raw_response_txt,\n            contexts=contexts,\n        )\n\n\nQueryResponseEvaluator = RelevancyEvaluator\n","sourceCodeStart":113,"sourceCodeEnd":145,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/relevancy.py#L113-L145","documentation":"RelevancyEvaluator.aevaluate judges relevancy by checking whether the LLM's synthesized answer contains 'yes' (lowercased substring). With raise_error=True a non-'yes' verdict is converted into ValueError('The response is invalid'); otherwise it becomes passing=False, score=0.0, with the raw answer kept as feedback.","triggerScenarios":"Evaluator constructed with raise_error=True; aevaluate returns a judge answer like 'No' or 'NO.' for an irrelevant context/response pair; judge returns empty/unparseable text lacking 'yes'.","commonSituations":"Strict eval configs meant to surface irrelevant retrievals as errors; substring matching surprises ('yesterday' contains 'yes'); judge refusals or empty completions flipping eval runs to hard failures.","solutions":["Set raise_error=False and branch on the returned EvaluationResult.passing instead","Review result.feedback to see the judge's raw answer when verdicts look wrong","If verdicts are systematically off, tune the eval template or judge model rather than crashing on 'no'"],"exampleFix":"# before\nevaluator = RelevancyEvaluator(llm=judge, raise_error=True)\nres = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)  # raises on 'No'\n\n# after\nevaluator = RelevancyEvaluator(llm=judge, raise_error=False)\nres = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)\nif not res.passing:\n    logger.info(\"irrelevant: %s\", res.feedback)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    res = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)\nexcept ValueError as e:\n    if str(e) == \"The response is invalid\":\n        res = EvaluationResult(query=q, response=resp, passing=False, score=0.0, feedback=\"judge said no\")\n    else:\n        raise","preventionTips":["Default to raise_error=False; use the returned score for gating logic","Watch for substring false-positives ('yes' inside other words) when interpreting feedback","Log feedback text on failures to catch judge drift early"],"tags":["evaluation","relevancy","llm-verdict","async","strict-mode"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}