open-mmlab/mmdetection · error · KeyError

{metric} is not in results

Error message

{metric} is not in results

What it means

During CocoMetric.compute_metrics, after converting predictions to COCO json files, the code looks up result_files[metric] for each metric being evaluated. If the per-metric result file was never produced (typically because the model returned no results of that type), it raises KeyError '{metric} is not in results'. It means the eval loop requested a metric whose predictions are absent from data_samples.

Source

Thrown at mmdet/evaluation/metrics/coco_metric.py:450

            logger.info(f'Evaluating {metric}...')

            # TODO: May refactor fast_eval_recall to an independent metric?
            # fast eval recall
            if metric == 'proposal_fast':
                ar = self.fast_eval_recall(
                    preds, self.proposal_nums, self.iou_thrs, logger=logger)
                log_msg = []
                for i, num in enumerate(self.proposal_nums):
                    eval_results[f'AR@{num}'] = ar[i]
                    log_msg.append(f'\nAR@{num}\t{ar[i]:.4f}')
                log_msg = ''.join(log_msg)
                logger.info(log_msg)
                continue

            # evaluate proposal, bbox and segm
            iou_type = 'bbox' if metric == 'proposal' else metric
            if metric not in result_files:
                raise KeyError(f'{metric} is not in results')
            try:
                predictions = load(result_files[metric])
                if iou_type == 'segm':
                    # Refer to https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L331  # noqa
                    # When evaluating mask AP, if the results contain bbox,
                    # cocoapi will use the box area instead of the mask area
                    # for calculating the instance area. Though the overall AP
                    # is not affected, this leads to different
                    # small/medium/large mask AP results.
                    for x in predictions:
                        x.pop('bbox')
                coco_dt = self._coco_api.loadRes(predictions)

            except IndexError:
                logger.error(
                    'The testing results of the whole dataset is empty.')
                break

View on GitHub (pinned to cfd5d3a985)

Solutions

1. Remove the unsupported metric from the metric list (e.g. drop 'segm' for a bbox-only model)
2. If you need 'segm', use a config/checkpoint with a mask head (e.g. Mask R-CNN) so predictions contain 'segm' results
3. Check data_samples contain the expected pred fields before eval (data_sample.pred_instances keys)

Example fix

# before (bbox-only model)
val_evaluator = dict(type='CocoMetric', metric=['bbox', 'segm'])
# after
val_evaluator = dict(type='CocoMetric', metric='bbox')

requested = {'bbox','segm'} if isinstance(metric, list) else {metric}
available = set(next(iter(data_samples)).get('pred_instances', {}).keys()) or set()
# inspect one sample's keys to see which result types the model emits
print('model produces:', available)

def model_supports(model_output_keys: set, wanted: str) -> bool:
    mapping = {'bbox': 'bboxes', 'segm': 'masks', 'proposal': 'proposals'}
    return mapping.get(wanted, 'bboxes') in model_output_keys or wanted == 'proposal_fast'

try:
    evaluator.compute_metrics(results)
except KeyError as e:
    missing = e.args[0].split(' is not in results')[0]
    print(f'model produced no {missing} predictions; drop it from metric list')

When it happens

Trigger: Setting metric=['bbox','segm'] while the model/detector only outputs detection boxes (no mask head), so no 'segm' result file is generated; or running segm eval on checkpoints trained without a mask branch; also 'proposal' eval when predictions lack proposal fields.

Common situations: Reusing a bbox-only config/checkpoint and just adding 'segm' to the evaluator; evaluating an RPN-only model with metric='bbox'; model outputs custom result keys that bypass the bbox2coco/segm2coco mapping.

Related errors

• metric should be one of 'bbox', 'segm', 'proposal', 'proposa
• metric item "{metric_item}" is not supported
• LoadImageFromFile is not found in the test pipeline
• Visualization needs the "visualizer" termdefined in the conf
• Unsupported input type: {type(single_input)}

AI-assisted analysis of open-mmlab/mmdetection@cfd5d3a985 (2026-08-27). Data as JSON: /api/errors/56b5000943e7a27c. Report an issue: GitHub.