run-llama/llama_index · error · ValueError

Retrieved texts must be provided

Error message

Retrieved texts must be provided

What it means

Raised by CohereRerankRelevancyMetric.compute when retrieved_texts is None. This metric scores relevancy by sending the retrieved chunks to the Cohere rerank API, so unlike id-based metrics (hit-rate, MRR, precision) it requires the actual text of each retrieved node; expected_texts is explicitly deleted as unused.

Source

Thrown at llama-index-core/llama_index/core/evaluation/retrieval/metrics.py:478

    def _get_agg_func(self, agg: Literal["max", "median", "mean"]) -> Callable:
        """Get agg func."""
        return _AGG_FUNC[agg]

    def compute(
        self,
        query: Optional[str] = None,
        expected_ids: Optional[List[str]] = None,
        retrieved_ids: Optional[List[str]] = None,
        expected_texts: Optional[List[str]] = None,
        retrieved_texts: Optional[List[str]] = None,
        agg: Literal["max", "median", "mean"] = "max",
        **kwargs: Any,
    ) -> RetrievalMetricResult:
        """Compute metric."""
        del expected_texts  # unused

        if retrieved_texts is None:
            raise ValueError("Retrieved texts must be provided")

        results = self._client.rerank(
            model=self.model,
            top_n=len(
                retrieved_texts
            ),  # i.e. get a rank score for each retrieved chunk
            query=query,
            documents=retrieved_texts,
        )
        relevance_scores = [r.relevance_score for r in results.results]
        agg_func = self._get_agg_func(agg)

        return RetrievalMetricResult(
            score=agg_func(relevance_scores), metadata={"agg": agg}
        )


METRIC_REGISTRY: Dict[str, Type[BaseRetrievalMetric]] = {

View on GitHub (pinned to afd0fef371)

Solutions

1. Construct the evaluator with RetrieverEvaluator(..., include_cohere_rerank=True) / include_retrieved_text=True so retrieved_texts is populated, or
2. Call compute() explicitly with retrieved_texts=[node.get_content() for node in retrieved_nodes].
3. Verify your metric list: if you cannot supply texts, use id-based metrics ('hit_rate', 'mrr', 'precision', 'recall', 'ap', 'ndcg') instead.

Example fix

# before
result = await metric.compute(query=q, retrieved_ids=ids)  # retrieved_texts omitted

# after
result = await metric.compute(
    query=q,
    retrieved_ids=ids,
    retrieved_texts=[n.get_content() for n in retrieved_nodes],
)

retrieved_texts = [n.get_content() for n in retrieved_nodes]
if not retrieved_texts or any(t is None for t in retrieved_texts):
    raise RuntimeError("retrieved_texts required for cohere_rerank_relevancy")

When it happens

Trigger: Calling RetrieverEvaluator with include_retrieved_text not enabled (so node texts are never populated), or calling metric.compute(retrieved_ids=[...]) without retrieved_texts — any evaluation run that supplies only ids where the cohere_rerank_relevancy metric is registered.

Common situations: Copying a RetrieverEvaluator setup from an example that used hit_rate/mrr and adding cohere_rerank_relevancy without also enabling text capture; programmatic eval harnesses that build kwargs generically and skip text fields.

Related errors

• Must specify both response and reference
• query and response must be provided
• query, contexts, and response must be provided
• Metric key {metric_key} not in results_df
• names and results_arr must have same length.

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/cc5c2ef3492b2b6d. Report an issue: GitHub.