run-llama/llama_index · error · ValueError

Metric key {metric_key} not in results_df

Error message

Metric key {metric_key} not in results_df

What it means

While aggregating retrieval metrics, get_retrieval_results_df iterates metric_keys and requires each key to be a column of the per-retriever DataFrame built from RetrievalEvalResult.metric_vals_dict. If a requested key never appears in those dicts, the column is missing and the function raises ValueError naming the offending key.

Source

Thrown at llama-index-core/llama_index/core/evaluation/notebook_utils.py:37

        import pandas as pd
    except ImportError:
        raise ImportError(
            "pandas is required for this function. Please install it with `pip install pandas`."
        )

    metric_keys = metric_keys or DEFAULT_METRIC_KEYS

    avg_metrics_dict = defaultdict(list)
    for name, eval_results in zip(names, results_arr):
        metric_dicts = []
        for eval_result in eval_results:
            metric_dict = eval_result.metric_vals_dict
            metric_dicts.append(metric_dict)
        results_df = pd.DataFrame(metric_dicts)

        for metric_key in metric_keys:
            if metric_key not in results_df.columns:
                raise ValueError(f"Metric key {metric_key} not in results_df")
            avg_metrics_dict[metric_key].append(results_df[metric_key].mean())

    return pd.DataFrame({"retrievers": names, **avg_metrics_dict})


def get_eval_results_df(
    names: List[str], results_arr: List[EvaluationResult], metric: Optional[str] = None
) -> Tuple[Any, Any]:
    """
    Organizes EvaluationResults into a deep dataframe and computes the mean
    score.

    result:
        result_df: pd.DataFrame representing all the evaluation results
        mean_df: pd.DataFrame of average scores groupby names
    """
    try:
        import pandas as pd

View on GitHub (pinned to afd0fef371)

Solutions

1. Align metric_keys with the metrics you actually computed: RetrieverEvaluator(metrics=['hit_rate'], ...)
2. Pass metric_keys explicitly derived from results: metric_keys = list(results[0][0].metric_vals_dict.keys())
3. Compute both default metrics (include mrr) or drop the key you don't have

Example fix

# before
evaluator = RetrieverEvaluator(metric=["hit_rate"], ...)
df = get_retrieval_results_df(names, results)  # default keys include 'mrr' -> raises

# after
df = get_retrieval_results_df(names, results, metric_keys=["hit_rate"])  # or compute mrr too

available_keys = set()
for eval_results in results_arr:
    for er in eval_results:
        available_keys.update(er.metric_vals_dict.keys())
safe_keys = [k for k in metric_keys if k in available_keys] if metric_keys else sorted(available_keys)
df = get_retrieval_results_df(names, results_arr, metric_keys=safe_keys)

try:
    df = get_retrieval_results_df(names, results, metric_keys=keys)
except ValueError as e:
    if "not in results_df" in str(e):
        missing = str(e).split('Metric key ')[-1].split(' not')[0]
        logger.warning("metric %s was not computed; rerun RetrieverEvaluator with it", missing)
    raise

When it happens

Trigger: Passing metric_keys=['mrr'] when results were produced only with a HitRate metric (metric_vals_dict lacks 'mrr'); requesting custom metric names that were never computed; mixing metric sets across retrievers so some columns are absent.

Common situations: RetrievalEvalMode/metrics changed between runs; default keys ['hit_rate','mrr'] assumed but only one metric was passed to RetrieverEvaluator(metrics=[...]); copy-pasted metric_keys from another project.

Related errors

• names and results_arr must have same length.
• Retrieved ids and expected ids must be provided
• query and response must be provided
• query, contexts, and response must be provided
• pandas is required for this function. Please install it with

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/04a79323bf2dab37. Report an issue: GitHub.