open-mmlab/mmdetection · error · RuntimeError

{k} is not a valid recall threshold

Error message

{k} is not a valid recall threshold

What it means

Flickr30kRecall add_positive raises RuntimeError when the recall threshold `k` is not one of the thresholds the metric was initialized with (i.e. not a key in total_byk_bycat, which is built from the topk list given at construction). It guards the per-k/per-category counters from being silently created for unknown thresholds.

Source

Thrown at mmdet/evaluation/metrics/flickr30k_metric.py:35

        """
        Parameters:
           - topk : tuple of ints corresponding to the recalls being
           tracked (eg, recall@1, recall@10, ...)
        """

        self.total_byk_bycat: Dict[int, Dict[str, int]] = {
            k: defaultdict(int)
            for k in topk
        }
        self.positives_byk_bycat: Dict[int, Dict[str, int]] = {
            k: defaultdict(int)
            for k in topk
        }

    def add_positive(self, k: int, category: str):
        """Log a positive hit @k for given category."""
        if k not in self.total_byk_bycat:
            raise RuntimeError(f'{k} is not a valid recall threshold')
        self.total_byk_bycat[k][category] += 1
        self.positives_byk_bycat[k][category] += 1

    def add_negative(self, k: int, category: str):
        """Log a negative hit @k for given category."""
        if k not in self.total_byk_bycat:
            raise RuntimeError(f'{k} is not a valid recall threshold')
        self.total_byk_bycat[k][category] += 1

    def report(self) -> Dict[str, Dict[str, float]]:
        """Return a condensed report of the results as a dict of dict.

        report[k][cat] is the recall@k for the given category
        """
        report: Dict[str, Dict[str, float]] = {}
        for k in self.total_byk_bycat:
            assert k in self.positives_byk_bycat
            report[str(k)] = {

View on GitHub (pinned to cfd5d3a985)

Solutions

1. Make the k values passed to add_positive exactly match the topk list given to the metric constructor
2. If you want additional thresholds, add them to the topk list when instantiating the metric
3. Ensure k is an int, not a string, before calling add_positive/add_negative

Example fix

// before
metric = Flickr30kMetric(topk=[1, 5, 10])
... metric.recall.add_positive(k=3, cat)  # raises
// after
metric = Flickr30kMetric(topk=[1, 3, 5, 10])
... metric.recall.add_positive(k=3, cat)

valid_ks = set(metric.recall.total_byk_bycat.keys())
if k not in valid_ks:
    raise ValueError(f'{k} not in configured topk {sorted(valid_ks)}')
metric.recall.add_positive(k, category)

def is_valid_recall_k(metric, k) -> bool:
    return k in metric.recall.total_byk_bycat

try:
    metric.recall.add_positive(k, category)
except RuntimeError:
    pass  # threshold not configured; skip or log

When it happens

Trigger: Calling add_positive(k, category) where k is not in the topk list passed to the Flickr30k metric constructor (e.g. metric built with topk=[1,5,10] but add_positive called with k=3), typically inside compute_metrics when scoring text-to-image retrieval results.

Common situations: Changing the retrieval code to report a different k than the configured topk list; passing k as a string ('5' vs 5); mismatch between the topk config used at metric construction and the ks the scoring loop iterates over.

Related errors

• metric should be one of 'bbox', 'segm', 'proposal', 'proposa
• metric {metric} is not supported.
• LoadImageFromFile is not found in the test pipeline
• Visualization needs the "visualizer" termdefined in the conf
• Unsupported input type: {type(single_input)}

AI-assisted analysis of open-mmlab/mmdetection@cfd5d3a985 (2026-08-27). Data as JSON: /api/errors/1e5ea56e64fcaca2. Report an issue: GitHub.