run-llama/llama_index · error · ValueError

The response is invalid

Error message

The response is invalid

What it means

Thrown by FaithfulnessEvaluator.aevaluate when raise_error=True and the LLM's verdict text does not contain the substring 'yes' (case-insensitive). The evaluator queries a SummaryIndex of the contexts with the response and treats any answer lacking 'yes' as a failed/unparseable verdict; with raise_error=False it returns passing=False and score=0.0 instead.

Source

Thrown at llama-index-core/llama_index/core/evaluation/faithfulness.py:192

        docs = [Document(text=context) for context in contexts]
        index = SummaryIndex.from_documents(docs)

        query_engine = index.as_query_engine(
            llm=self._llm,
            text_qa_template=self._eval_template,
            refine_template=self._refine_template,
        )
        response_obj = await query_engine.aquery(response)

        raw_response_txt = str(response_obj)

        if "yes" in raw_response_txt.lower():
            passing = True
        else:
            passing = False
            if self._raise_error:
                raise ValueError("The response is invalid")

        return EvaluationResult(
            query=query,
            response=response,
            contexts=contexts,
            passing=passing,
            score=1.0 if passing else 0.0,
            feedback=raw_response_txt,
        )


# legacy: backward compatibility
ResponseEvaluator = FaithfulnessEvaluator

View on GitHub (pinned to afd0fef371)

Solutions

1. If a genuine 'No' verdict is expected behavior, use raise_error=False and branch on result.passing instead of catching exceptions.
2. Use a stronger LLM / raise token limits so the model follows the YES/NO format from the default template.
3. Inspect result.feedback (with raise_error=False) to distinguish real 'No' verdicts from unparseable output, then tune the eval/refine templates.

Example fix

# before
evaluator = FaithfulnessEvaluator(llm=llm, raise_error=True)
result = await evaluator.aevaluate(response=r, contexts=ctxs)  # raises on any non-yes

# after
evaluator = FaithfulnessEvaluator(llm=llm, raise_error=False)
result = await evaluator.aevaluate(response=r, contexts=ctxs)
if not result.passing:
    logger.info("unfaithful or unparseable: %s", result.feedback)

result = await evaluator.aevaluate(response=r, contexts=ctxs)
if not result.passing:
    logger.info("faithfulness verdict feedback: %s", result.feedback)

def is_faithful(r) -> bool:
    return bool(r.passing)

try:
    result = await evaluator.aevaluate(response=r, contexts=ctxs)
except ValueError as e:
    if e.args[0] == "The response is invalid" and strict:
        return EvaluationResult-like fallback
    raise

When it happens

Trigger: await evaluator.aevaluate(response=..., contexts=[...]) with raise_error=True where the LLM answers 'No.', 'NO', explains why the response is unsupported, returns empty output, or the verdict got truncated.

Common situations: Small/local models returning verbose explanations instead of YES/NO; empty completions from token limits; models answering in another language; raise_error=True set for strict pipelines where any non-yes verdict aborts the batch.

Related errors

• The response is invalid
• The response is invalid
• The response is invalid
• contexts and response must be provided
• The response is invalid

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/0e6e4af7420c79a4. Report an issue: GitHub.