run-llama/llama_index · error · ValueError

The response is invalid

Error message

The response is invalid

What it means

RelevancyEvaluator.aevaluate judges relevancy by checking whether the LLM's synthesized answer contains 'yes' (lowercased substring). With raise_error=True a non-'yes' verdict is converted into ValueError('The response is invalid'); otherwise it becomes passing=False, score=0.0, with the raw answer kept as feedback.

Source

Thrown at llama-index-core/llama_index/core/evaluation/relevancy.py:131

        query_response = f"Question: {query}\nResponse: {response}"

        await asyncio.sleep(sleep_time_in_seconds)

        query_engine = index.as_query_engine(
            llm=self._llm,
            text_qa_template=self._eval_template,
            refine_template=self._refine_template,
        )
        response_obj = await query_engine.aquery(query_response)

        raw_response_txt = str(response_obj)

        if "yes" in raw_response_txt.lower():
            passing = True
        else:
            if self._raise_error:
                raise ValueError("The response is invalid")
            passing = False

        return EvaluationResult(
            query=query,
            response=response,
            passing=passing,
            score=1.0 if passing else 0.0,
            feedback=raw_response_txt,
            contexts=contexts,
        )


QueryResponseEvaluator = RelevancyEvaluator

View on GitHub (pinned to afd0fef371)

Solutions

1. Set raise_error=False and branch on the returned EvaluationResult.passing instead
2. Review result.feedback to see the judge's raw answer when verdicts look wrong
3. If verdicts are systematically off, tune the eval template or judge model rather than crashing on 'no'

Example fix

# before
evaluator = RelevancyEvaluator(llm=judge, raise_error=True)
res = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)  # raises on 'No'

# after
evaluator = RelevancyEvaluator(llm=judge, raise_error=False)
res = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)
if not res.passing:
    logger.info("irrelevant: %s", res.feedback)

try:
    res = await evaluator.aevaluate(query=q, contexts=ctx, response=resp)
except ValueError as e:
    if str(e) == "The response is invalid":
        res = EvaluationResult(query=q, response=resp, passing=False, score=0.0, feedback="judge said no")
    else:
        raise

When it happens

Trigger: Evaluator constructed with raise_error=True; aevaluate returns a judge answer like 'No' or 'NO.' for an irrelevant context/response pair; judge returns empty/unparseable text lacking 'yes'.

Common situations: Strict eval configs meant to surface irrelevant retrievals as errors; substring matching surprises ('yesterday' contains 'yes'); judge refusals or empty completions flipping eval runs to hard failures.

Related errors

• The response is invalid
• query, contexts, and response must be provided
• The response is invalid
• Impossible score results. Total amount of votes is 2.
• query, response, second_response, and reference must be prov

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/dd1190cd6c4e1204. Report an issue: GitHub.