open-mmlab/mmdetection · error · KeyError

metric should be one of 'MR', 'AP', 'JI',but got {metric}.

Error message

metric should be one of 'MR', 'AP', 'JI',but got {metric}.

What it means

CrowdHumanMetric.__init__ validates the `metric` argument against CrowdHuman's supported metrics: 'MR' (log-average miss rate), 'AP', 'JI' (Jaccard Index). Any other value raises KeyError. This is the CrowdHuman analogue of CocoMetric's metric whitelist and fails at evaluator construction, before any eval runs.

Source

Thrown at mmdet/evaluation/metrics/crowdhuman_metric.py:89

                 outfile_prefix: Optional[str] = None,
                 file_client_args: dict = None,
                 backend_args: dict = None,
                 collect_device: str = 'cpu',
                 prefix: Optional[str] = None,
                 eval_mode: int = 0,
                 iou_thres: float = 0.5,
                 compare_matching_method: Optional[str] = None,
                 mr_ref: str = 'CALTECH_-2',
                 num_ji_process: int = 10) -> None:
        super().__init__(collect_device=collect_device, prefix=prefix)

        self.ann_file = ann_file
        # crowdhuman evaluation metrics
        self.metrics = metric if isinstance(metric, list) else [metric]
        allowed_metrics = ['MR', 'AP', 'JI']
        for metric in self.metrics:
            if metric not in allowed_metrics:
                raise KeyError(f"metric should be one of 'MR', 'AP', 'JI',"
                               f'but got {metric}.')

        self.format_only = format_only
        if self.format_only:
            assert outfile_prefix is not None, 'outfile_prefix must be not'
            'None when format_only is True, otherwise the result files will'
            'be saved to a temp directory which will be cleaned up at the end.'
        self.outfile_prefix = outfile_prefix
        self.backend_args = backend_args
        if file_client_args is not None:
            raise RuntimeError(
                'The `file_client_args` is deprecated, '
                'please use `backend_args` instead, please refer to'
                'https://github.com/open-mmlab/mmdetection/blob/main/configs/_base_/datasets/coco_detection.py'  # noqa: E501
            )

        assert eval_mode in [0, 1, 2], \
            "Unknown eval mode. mr_ref should be one of '0', '1', '2'."

View on GitHub (pinned to cfd5d3a985)

Solutions

1. Use only 'MR', 'AP', 'JI' or a list of them, e.g. dict(type='CrowdHumanMetric', metric=['MR','AP','JI'])
2. If you wanted COCO metrics, you are using the wrong metric class — switch to CocoMetric
3. Double-check the dataset type matches the evaluator type in the config

Example fix

# before
val_evaluator = dict(type='CrowdHumanMetric', ann_file=..., metric='bbox')
# after
val_evaluator = dict(type='CrowdHumanMetric', ann_file=..., metric=['AP','MR','JI'])

ALLOWED = {'MR', 'AP', 'JI'}
metrics = cfg['val_evaluator']['metric']
metrics = [metrics] if isinstance(metrics, str) else metrics
bad = [m for m in metrics if m not in ALLOWED]
assert not bad, f'Invalid CrowdHumanMetric metric(s): {bad}; allowed: MR/AP/JI'

def is_valid_crowdhuman_metric(m) -> bool:
    items = [m] if isinstance(m, str) else m
    return bool(items) and all(x in {'MR', 'AP', 'JI'} for x in items)

When it happens

Trigger: Passing metric='bbox' (a COCO name) to CrowdHumanMetric, or misspelled/lowercase names like 'ap'/'ji '; mixing metric lists from COCO configs into a CrowdHuman config.

Common situations: Adapting a COCO detection config to CrowdHuman and leaving metric=['bbox']; copy-paste between evaluator types; assuming mmdet metric names are universal across datasets.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• metric should be one of 'bbox', 'segm', 'proposal', 'proposa
• metric item "{metric_item}" is not supported
• The `file_client_args` is deprecated, please use `backend_ar
• LoadImageFromFile is not found in the test pipeline
• Visualization needs the "visualizer" termdefined in the conf

AI-assisted analysis of open-mmlab/mmdetection@cfd5d3a985 (2026-08-27). Data as JSON: /api/errors/7740cfddcc4dff55. Report an issue: GitHub.