run-llama/llama_index · error · ValueError

Must specify both response and reference

Error message

Must specify both response and reference

What it means

Raised by SemanticSimilarityEvaluator.aevaluate when either response or reference is None. This evaluator embeds the generated response and a ground-truth reference and compares the embeddings; query and contexts are explicitly deleted as unused, so both text arguments are mandatory inputs.

Source

Thrown at llama-index-core/llama_index/core/evaluation/semantic_similarity.py:70

    def _get_prompts(self) -> PromptDictType:
        """Get prompts."""
        return {}

    def _update_prompts(self, prompts: PromptDictType) -> None:
        """Update prompts."""

    async def aevaluate(
        self,
        query: Optional[str] = None,
        response: Optional[str] = None,
        contexts: Optional[Sequence[str]] = None,
        reference: Optional[str] = None,
        **kwargs: Any,
    ) -> EvaluationResult:
        del query, contexts, kwargs  # Unused

        if response is None or reference is None:
            raise ValueError("Must specify both response and reference")

        response_embedding = await self._embed_model.aget_text_embedding(response)
        reference_embedding = await self._embed_model.aget_text_embedding(reference)

        similarity_score = self._similarity_fn(response_embedding, reference_embedding)
        passing = similarity_score >= self._similarity_threshold
        return EvaluationResult(
            score=similarity_score,
            passing=passing,
            feedback=f"Similarity score: {similarity_score}",
        )

View on GitHub (pinned to afd0fef371)

Solutions

1. Provide both strings: await evaluator.aevaluate(response=resp, reference=ref).
2. Skip or separately flag dataset rows with missing references before running evaluation.
3. Guard upstream: if the LLM response is None/empty, short-circuit the eval with a failing result instead of calling aevaluate.

Example fix

# before
result = await evaluator.aevaluate(query=q, contexts=ctx)  # response/reference missing

# after
result = await evaluator.aevaluate(response=answer, reference=gold_answer)

if not response or not reference:
    return EvaluationResult(score=0.0, passing=False, feedback="missing response/reference")
result = await evaluator.aevaluate(response=response, reference=reference)

When it happens

Trigger: Calling await evaluator.aevaluate(query=..., contexts=...) without response/reference; passing reference=None for LLM-generated answers with no gold answer; evaluating a pipeline whose response was empty (None) due to an upstream failure.

Common situations: Building an eval harness that loops over datasets where some rows lack reference answers; an upstream LLM call returning None and the harness forwarding it; reusing a CorrectnessEvaluator-style call signature that passes query+contexts only.

Related errors

• Retrieved texts must be provided
• query and response must be provided
• query, contexts, and response must be provided
• Metric key {metric_key} not in results_df
• names and results_arr must have same length.

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/ff6cfe5748fb6504. Report an issue: GitHub.