run-llama/llama_index · error · ValueError

The response is invalid

Error message

The response is invalid

What it means

Thrown by ContextRelevancyEvaluator.aevaluate when raise_error=True and the parser_function returned None for both score and reasoning from the LLM's answer. The evaluator runs the query against a SummaryIndex built from the contexts and parses the raw response text, so any non-conforming LLM answer triggers this.

Source

Thrown at llama-index-core/llama_index/core/evaluation/context_relevancy.py:163

        docs = [Document(text=context) for context in contexts]
        index = SummaryIndex.from_documents(docs)

        await asyncio.sleep(sleep_time_in_seconds)

        query_engine = index.as_query_engine(
            llm=self._llm,
            text_qa_template=self._eval_template,
            refine_template=self._refine_template,
        )
        response_obj = await query_engine.aquery(query)
        raw_response_txt = str(response_obj)

        score, reasoning = self.parser_function(raw_response_txt)

        invalid_result, invalid_reason = False, None
        if score is None and reasoning is None:
            if self._raise_error:
                raise ValueError("The response is invalid")
            invalid_result = True
            invalid_reason = "Unable to parse the output string."

        if score:
            score /= self.score_threshold

        return EvaluationResult(
            query=query,
            contexts=contexts,
            score=score,
            feedback=raw_response_txt,
            invalid_result=invalid_result,
            invalid_reason=invalid_reason,
        )

View on GitHub (pinned to afd0fef371)

Solutions

1. Run once with raise_error=False and inspect EvaluationResult.feedback to see the raw model output.
2. Use a stronger LLM or increase its token limit so the formatted answer completes.
3. Customize eval_template/refine_template (or parser_function) so output reliably matches the parseable format.
4. Set raise_error=False and handle invalid_result=True results in your pipeline.

Example fix

# before
evaluator = ContextRelevancyEvaluator(llm=llm, raise_error=True)

# after
evaluator = ContextRelevancyEvaluator(llm=llm, raise_error=False)
result = await evaluator.aevaluate(query=q, contexts=ctxs)
if result.invalid_result:
    logger.warning("unparseable context relevancy output: %s", result.feedback)

result = await evaluator.aevaluate(query=q, contexts=ctxs)
if result.invalid_result:
    # parser failed; feedback holds the raw LLM text
    ...

def valid_context_relevancy(r) -> bool:
    return not r.invalid_result and r.score is not None

try:
    result = await evaluator.aevaluate(query=q, contexts=ctxs)
except ValueError as e:
    if e.args[0] == "The response is invalid":
        return None
    raise

When it happens

Trigger: await evaluator.aevaluate(query=..., contexts=[...]) with raise_error=True where the LLM (driven by the eval/refine templates) replies with prose, an empty string, or a truncated answer that the parser cannot extract a score from.

Common situations: Weak or small LLMs ignoring the YES/NO-plus-score format expected by the default template; low max_tokens truncating output; refine steps producing text that strays from the format; custom templates without a matching parser_function.

Related errors

• The response is invalid
• Both query and contexts must be provided
• The response is invalid
• Invalid JSON Path: {expression}
• No valid JSON found in output: {output}

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/62f3c4f2f06d65fb. Report an issue: GitHub.