{"record":{"id":"62f3c4f2f06d65fb","repo":"run-llama/llama_index","slug":"the-response-is-invalid-62f3c4","errorCode":null,"errorMessage":"The response is invalid","messagePattern":"The response is invalid","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/context_relevancy.py","lineNumber":163,"sourceCode":"        docs = [Document(text=context) for context in contexts]\n        index = SummaryIndex.from_documents(docs)\n\n        await asyncio.sleep(sleep_time_in_seconds)\n\n        query_engine = index.as_query_engine(\n            llm=self._llm,\n            text_qa_template=self._eval_template,\n            refine_template=self._refine_template,\n        )\n        response_obj = await query_engine.aquery(query)\n        raw_response_txt = str(response_obj)\n\n        score, reasoning = self.parser_function(raw_response_txt)\n\n        invalid_result, invalid_reason = False, None\n        if score is None and reasoning is None:\n            if self._raise_error:\n                raise ValueError(\"The response is invalid\")\n            invalid_result = True\n            invalid_reason = \"Unable to parse the output string.\"\n\n        if score:\n            score /= self.score_threshold\n\n        return EvaluationResult(\n            query=query,\n            contexts=contexts,\n            score=score,\n            feedback=raw_response_txt,\n            invalid_result=invalid_result,\n            invalid_reason=invalid_reason,\n        )\n","sourceCodeStart":145,"sourceCodeEnd":178,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/context_relevancy.py#L145-L178","documentation":"Thrown by ContextRelevancyEvaluator.aevaluate when raise_error=True and the parser_function returned None for both score and reasoning from the LLM's answer. The evaluator runs the query against a SummaryIndex built from the contexts and parses the raw response text, so any non-conforming LLM answer triggers this.","triggerScenarios":"await evaluator.aevaluate(query=..., contexts=[...]) with raise_error=True where the LLM (driven by the eval/refine templates) replies with prose, an empty string, or a truncated answer that the parser cannot extract a score from.","commonSituations":"Weak or small LLMs ignoring the YES/NO-plus-score format expected by the default template; low max_tokens truncating output; refine steps producing text that strays from the format; custom templates without a matching parser_function.","solutions":["Run once with raise_error=False and inspect EvaluationResult.feedback to see the raw model output.","Use a stronger LLM or increase its token limit so the formatted answer completes.","Customize eval_template/refine_template (or parser_function) so output reliably matches the parseable format.","Set raise_error=False and handle invalid_result=True results in your pipeline."],"exampleFix":"# before\nevaluator = ContextRelevancyEvaluator(llm=llm, raise_error=True)\n\n# after\nevaluator = ContextRelevancyEvaluator(llm=llm, raise_error=False)\nresult = await evaluator.aevaluate(query=q, contexts=ctxs)\nif result.invalid_result:\n    logger.warning(\"unparseable context relevancy output: %s\", result.feedback)","handlingStrategy":"fallback","validationCode":"result = await evaluator.aevaluate(query=q, contexts=ctxs)\nif result.invalid_result:\n    # parser failed; feedback holds the raw LLM text\n    ...","typeGuard":"def valid_context_relevancy(r) -> bool:\n    return not r.invalid_result and r.score is not None","tryCatchPattern":"try:\n    result = await evaluator.aevaluate(query=q, contexts=ctxs)\nexcept ValueError as e:\n    if e.args[0] == \"The response is invalid\":\n        return None\n    raise","preventionTips":["Keep raise_error=False unless every result must parse; then verify the model honors the template format first.","Check EvaluationResult.feedback whenever scores look wrong — format drift shows up there before errors do."],"tags":["llm-output","parsing","evaluation","context-relevancy"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}