run-llama/llama_index · error · ValueError

Retrieved ids and expected ids must be provided

Error message

Retrieved ids and expected ids must be provided

What it means

HitRate.compute requires both retrieved_ids and expected_ids to be non-None AND non-empty. The guard rejects None or falsy lists (empty lists fail `not retrieved_ids`), so evaluating a retrieval that returned zero docs raises ValueError before scoring.

Source

Thrown at llama-index-core/llama_index/core/evaluation/retrieval/metrics.py:68

            retrieved_texts (Optional[List[str]]): Retrieved texts (not used in the current implementation).

        Raises
        ------
            ValueError: If the necessary IDs are not provided.

        Returns
        -------
            RetrievalMetricResult: The result with the computed hit rate score.

        """
        # Checking for the required arguments
        if (
            retrieved_ids is None
            or expected_ids is None
            or not retrieved_ids
            or not expected_ids
        ):
            raise ValueError("Retrieved ids and expected ids must be provided")

        if self.use_granular_hit_rate:
            # Granular HitRate calculation: Calculate all hits and divide by the number of expected docs
            expected_set = set(expected_ids)
            hits = sum(1 for doc_id in retrieved_ids if doc_id in expected_set)
            score = hits / len(expected_ids) if expected_ids else 0.0
        else:
            # Default HitRate calculation: Check if there is a single hit
            is_hit = any(id in expected_ids for id in retrieved_ids)
            score = 1.0 if is_hit else 0.0

        return RetrievalMetricResult(score=score)


class MRR(BaseRetrievalMetric):
    """
    MRR (Mean Reciprocal Rank) metric with two calculation options.

View on GitHub (pinned to afd0fef371)

Solutions

1. Skip metric computation when either list is empty rather than calling compute
2. Ensure your retriever returns at least one node (check similarity cutoffs / filters that can zero out results)
3. Validate the dataset: every query must have at least one expected id

Example fix

# before
score = hit_rate.compute(retrieved_ids=retrieved, expected_ids=expected)  # retrieved == []

# after
if retrieved and expected:
    score = hit_rate.compute(retrieved_ids=retrieved, expected_ids=expected)
else:
    score = 0.0  # or skip this query

def compute_hit_rate(metric, retrieved_ids, expected_ids):
    if not retrieved_ids or not expected_ids:
        return RetrievalMetricResult(score=0.0)  # or None to mark not-computable
    return metric.compute(retrieved_ids=retrieved_ids, expected_ids=expected_ids)

When it happens

Trigger: Calling hit_rate.compute(retrieved_ids=[], expected_ids=['id1']) (retriever returned nothing), or either argument None, or both empty.

Common situations: Top-k retrievers returning zero results for out-of-domain queries; eval datasets with empty golden ids; pipeline bugs producing empty retrieved lists; treating empty retrieval as scoreable.

Related errors

• query, contexts, and response must be provided
• Metric key {metric_key} not in results_df
• names and results_arr must have same length.
• query, response, second_response, and reference must be prov
• query, contexts, and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/d795587e81d26b01. Report an issue: GitHub.