{"record":{"id":"046a6b443472dd3c","repo":"run-llama/llama_index","slug":"the-response-is-invalid","errorCode":null,"errorMessage":"The response is invalid","messagePattern":"The response is invalid","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/answer_relevancy.py","lineNumber":132,"sourceCode":"        del contexts  # Unused\n\n        if query is None or response is None:\n            raise ValueError(\"query and response must be provided\")\n\n        await asyncio.sleep(sleep_time_in_seconds)\n\n        eval_response = await self._llm.apredict(\n            prompt=self._eval_template,\n            query=query,\n            response=response,\n        )\n\n        score, reasoning = self.parser_function(eval_response)\n\n        invalid_result, invalid_reason = False, None\n        if score is None and reasoning is None:\n            if self._raise_error:\n                raise ValueError(\"The response is invalid\")\n            invalid_result = True\n            invalid_reason = \"Unable to parse the output string.\"\n\n        if score:\n            score /= self.score_threshold\n\n        return EvaluationResult(\n            query=query,\n            response=response,\n            score=score,\n            feedback=eval_response,\n            invalid_result=invalid_result,\n            invalid_reason=invalid_reason,\n        )\n","sourceCodeStart":114,"sourceCodeEnd":147,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/answer_relevancy.py#L114-L147","documentation":"Thrown by AnswerRelevancyEvaluator.aevaluate when raise_error=True and the evaluator's parser_function could not extract either a score or reasoning from the LLM's evaluation output (both are None). It indicates the LLM returned text that does not match the expected format rather than a bug in your code. With raise_error=False the same condition instead produces EvaluationResult(invalid_result=True).","triggerScenarios":"Calling await evaluator.aevaluate(query=..., response=...) (or running it via BatchRunner) with raise_error=True, where the configured LLM answers the relevancy prompt with free-form text, empty output, or a refusal instead of the parseable score/reasoning format the default parser expects.","commonSituations":"Using a small/local model (e.g. Llama or a small OpenAI model) that ignores the output format; max_tokens too low so output is truncated before the score; a custom eval_template whose output the default parser_function cannot parse; JSON-mode-off LLMs returning prose.","solutions":["Inspect the EvaluationResult.feedback / raw LLM output (run once with raise_error=False) to see exactly what the model returned and adjust the prompt or model accordingly.","Pass a stronger LLM (e.g. gpt-4 class) or raise max_output_tokens so the full formatted answer is produced.","Supply a custom parser_function that matches your template's output format, or override the default eval_template so the model emits the expected structure.","If unparseable outputs are acceptable in your pipeline, construct the evaluator with raise_error=False and filter on EvaluationResult.invalid_result afterwards."],"exampleFix":"// before\nevaluator = AnswerRelevancyEvaluator(llm=llm, raise_error=True)\nresult = await evaluator.aevaluate(query=q, response=r)  # raises on unparseable output\n\n// after\nevaluator = AnswerRelevancyEvaluator(llm=llm, raise_error=False)\nresult = await evaluator.aevaluate(query=q, response=r)\nif result.invalid_result:\n    # log result.feedback and handle gracefully\n    ...","handlingStrategy":"fallback","validationCode":"from llama_index.core.evaluation import EvaluationResult\n\ndef is_parseable(result: EvaluationResult) -> bool:\n    return not result.invalid_result","typeGuard":"def has_valid_score(r) -> bool:\n    return r is not None and not r.invalid_result and r.score is not None","tryCatchPattern":"try:\n    result = await evaluator.aevaluate(query=q, response=r)\nexcept ValueError as e:\n    if e.args[0] == \"The response is invalid\":\n        logger.warning(\"unparseable relevancy output for query=%s\", q)\n        return None  # or retry with a stronger LLM\n    raise","preventionTips":["Run AnswerRelevancyEvaluator with raise_error=False by default; only enable strict mode once outputs parse reliably.","Pin a model known to follow the output format and set adequate max_output_tokens.","Log result.feedback for invalid_result=True items to catch prompt-format drift early."],"tags":["llm-output","parsing","evaluation","answer-relevancy"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}