run-llama/llama_index · error · ValueError

The response is invalid

Error message

The response is invalid

What it means

Thrown by AnswerRelevancyEvaluator.aevaluate when raise_error=True and the evaluator's parser_function could not extract either a score or reasoning from the LLM's evaluation output (both are None). It indicates the LLM returned text that does not match the expected format rather than a bug in your code. With raise_error=False the same condition instead produces EvaluationResult(invalid_result=True).

Source

Thrown at llama-index-core/llama_index/core/evaluation/answer_relevancy.py:132

        del contexts  # Unused

        if query is None or response is None:
            raise ValueError("query and response must be provided")

        await asyncio.sleep(sleep_time_in_seconds)

        eval_response = await self._llm.apredict(
            prompt=self._eval_template,
            query=query,
            response=response,
        )

        score, reasoning = self.parser_function(eval_response)

        invalid_result, invalid_reason = False, None
        if score is None and reasoning is None:
            if self._raise_error:
                raise ValueError("The response is invalid")
            invalid_result = True
            invalid_reason = "Unable to parse the output string."

        if score:
            score /= self.score_threshold

        return EvaluationResult(
            query=query,
            response=response,
            score=score,
            feedback=eval_response,
            invalid_result=invalid_result,
            invalid_reason=invalid_reason,
        )

View on GitHub (pinned to afd0fef371)

Solutions

1. Inspect the EvaluationResult.feedback / raw LLM output (run once with raise_error=False) to see exactly what the model returned and adjust the prompt or model accordingly.
2. Pass a stronger LLM (e.g. gpt-4 class) or raise max_output_tokens so the full formatted answer is produced.
3. Supply a custom parser_function that matches your template's output format, or override the default eval_template so the model emits the expected structure.
4. If unparseable outputs are acceptable in your pipeline, construct the evaluator with raise_error=False and filter on EvaluationResult.invalid_result afterwards.

Example fix

// before
evaluator = AnswerRelevancyEvaluator(llm=llm, raise_error=True)
result = await evaluator.aevaluate(query=q, response=r)  # raises on unparseable output

// after
evaluator = AnswerRelevancyEvaluator(llm=llm, raise_error=False)
result = await evaluator.aevaluate(query=q, response=r)
if result.invalid_result:
    # log result.feedback and handle gracefully
    ...

from llama_index.core.evaluation import EvaluationResult

def is_parseable(result: EvaluationResult) -> bool:
    return not result.invalid_result

def has_valid_score(r) -> bool:
    return r is not None and not r.invalid_result and r.score is not None

try:
    result = await evaluator.aevaluate(query=q, response=r)
except ValueError as e:
    if e.args[0] == "The response is invalid":
        logger.warning("unparseable relevancy output for query=%s", q)
        return None  # or retry with a stronger LLM
    raise

When it happens

Trigger: Calling await evaluator.aevaluate(query=..., response=...) (or running it via BatchRunner) with raise_error=True, where the configured LLM answers the relevancy prompt with free-form text, empty output, or a refusal instead of the parseable score/reasoning format the default parser expects.

Common situations: Using a small/local model (e.g. Llama or a small OpenAI model) that ignores the output format; max_tokens too low so output is truncated before the score; a custom eval_template whose output the default parser_function cannot parse; JSON-mode-off LLMs returning prose.

Related errors

• The response is invalid
• The response is invalid
• Invalid JSON Path: {expression}
• No valid JSON found in output: {output}
• Unknown tool name: {tool_call_result.tool_name}

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/046a6b443472dd3c. Report an issue: GitHub.