run-llama/llama_index · error · ValueError

Invalid metric name: {metric}

Error message

Invalid metric name: {metric}

What it means

Raised by resolve_metrics() in the retrieval-evaluation metrics registry when a requested metric name is not a key of METRIC_REGISTRY. Valid names are exactly: hit_rate, mrr (plus their classes), precision, recall, ap, ndcg, and cohere_rerank_relevancy; the check is a simple dict membership test before class resolution.

Source

Thrown at llama-index-core/llama_index/core/evaluation/retrieval/metrics.py:511

        )


METRIC_REGISTRY: Dict[str, Type[BaseRetrievalMetric]] = {
    "hit_rate": HitRate,
    "mrr": MRR,
    "precision": Precision,
    "recall": Recall,
    "ap": AveragePrecision,
    "ndcg": NDCG,
    "cohere_rerank_relevancy": CohereRerankRelevancyMetric,
}


def resolve_metrics(metrics: List[str]) -> List[Type[BaseRetrievalMetric]]:
    """Resolve metrics from list of metric names."""
    for metric in metrics:
        if metric not in METRIC_REGISTRY:
            raise ValueError(f"Invalid metric name: {metric}")

    return [METRIC_REGISTRY[metric] for metric in metrics]

View on GitHub (pinned to afd0fef371)

Solutions

1. Use one of the registered names: 'hit_rate', 'mrr', 'precision', 'recall', 'ap', 'ndcg', 'cohere_rerank_relevancy'.
2. Validate user-provided metric names against METRIC_REGISTRY keys before calling resolve_metrics, and surface the allowed list in your error message.
3. Check for hyphen/underscore mismatches and case differences in config-driven metric lists.

Example fix

# before
metrics = resolve_metrics(['hit-rate', 'average_precision'])

# after
from llama_index.core.evaluation.retrieval.metrics import METRIC_REGISTRY
metrics = resolve_metrics(['hit_rate', 'ap'])
assert set(user_metrics) <= set(METRIC_REGISTRY), f"allowed: {sorted(METRIC_REGISTRY)}"

from llama_index.core.evaluation.retrieval.metrics import METRIC_REGISTRY

invalid = [m for m in requested_metrics if m not in METRIC_REGISTRY]
if invalid:
    raise ValueError(f"Unknown metrics {invalid}; allowed: {sorted(METRIC_REGISTRY)}")
metrics = resolve_metrics(requested_metrics)

When it happens

Trigger: Calling resolve_metrics(['hit-rate']) with a hyphen instead of underscore, passing a metric class instead of its name string, or passing a typo like 'cohire_rerank_relevancy' when building a RetrieverEvaluator.

Common situations: Translating metric names from other libraries (sklearn's 'average_precision', BEIR conventions); renaming between llama-index versions; user-supplied config files (YAML/CLI) feeding metric names into an eval harness.

Related errors

• Cannot specify both similarity_fn and similarity_mode
• Root agent {root_agent} not found in provided agents
• query and response must be provided
• query, contexts, and response must be provided
• Metric key {metric_key} not in results_df

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/9ce3399abf27e32c. Report an issue: GitHub.