{"record":{"id":"0e6e4af7420c79a4","repo":"run-llama/llama_index","slug":"the-response-is-invalid-0e6e4a","errorCode":null,"errorMessage":"The response is invalid","messagePattern":"The response is invalid","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/faithfulness.py","lineNumber":192,"sourceCode":"\n        docs = [Document(text=context) for context in contexts]\n        index = SummaryIndex.from_documents(docs)\n\n        query_engine = index.as_query_engine(\n            llm=self._llm,\n            text_qa_template=self._eval_template,\n            refine_template=self._refine_template,\n        )\n        response_obj = await query_engine.aquery(response)\n\n        raw_response_txt = str(response_obj)\n\n        if \"yes\" in raw_response_txt.lower():\n            passing = True\n        else:\n            passing = False\n            if self._raise_error:\n                raise ValueError(\"The response is invalid\")\n\n        return EvaluationResult(\n            query=query,\n            response=response,\n            contexts=contexts,\n            passing=passing,\n            score=1.0 if passing else 0.0,\n            feedback=raw_response_txt,\n        )\n\n\n# legacy: backward compatibility\nResponseEvaluator = FaithfulnessEvaluator\n","sourceCodeStart":174,"sourceCodeEnd":206,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/faithfulness.py#L174-L206","documentation":"Thrown by FaithfulnessEvaluator.aevaluate when raise_error=True and the LLM's verdict text does not contain the substring 'yes' (case-insensitive). The evaluator queries a SummaryIndex of the contexts with the response and treats any answer lacking 'yes' as a failed/unparseable verdict; with raise_error=False it returns passing=False and score=0.0 instead.","triggerScenarios":"await evaluator.aevaluate(response=..., contexts=[...]) with raise_error=True where the LLM answers 'No.', 'NO', explains why the response is unsupported, returns empty output, or the verdict got truncated.","commonSituations":"Small/local models returning verbose explanations instead of YES/NO; empty completions from token limits; models answering in another language; raise_error=True set for strict pipelines where any non-yes verdict aborts the batch.","solutions":["If a genuine 'No' verdict is expected behavior, use raise_error=False and branch on result.passing instead of catching exceptions.","Use a stronger LLM / raise token limits so the model follows the YES/NO format from the default template.","Inspect result.feedback (with raise_error=False) to distinguish real 'No' verdicts from unparseable output, then tune the eval/refine templates."],"exampleFix":"# before\nevaluator = FaithfulnessEvaluator(llm=llm, raise_error=True)\nresult = await evaluator.aevaluate(response=r, contexts=ctxs)  # raises on any non-yes\n\n# after\nevaluator = FaithfulnessEvaluator(llm=llm, raise_error=False)\nresult = await evaluator.aevaluate(response=r, contexts=ctxs)\nif not result.passing:\n    logger.info(\"unfaithful or unparseable: %s\", result.feedback)","handlingStrategy":"fallback","validationCode":"result = await evaluator.aevaluate(response=r, contexts=ctxs)\nif not result.passing:\n    logger.info(\"faithfulness verdict feedback: %s\", result.feedback)","typeGuard":"def is_faithful(r) -> bool:\n    return bool(r.passing)","tryCatchPattern":"try:\n    result = await evaluator.aevaluate(response=r, contexts=ctxs)\nexcept ValueError as e:\n    if e.args[0] == \"The response is invalid\" and strict:\n        return EvaluationResult-like fallback\n    raise","preventionTips":["Reserve raise_error=True for smoke tests; production batch runs should branch on result.passing.","Confirm the LLM answers the default template with YES/NO before enabling strict mode."],"tags":["llm-output","faithfulness","evaluation","strict-mode"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}