{"record":{"id":"7740cfddcc4dff55","repo":"open-mmlab/mmdetection","slug":"metric-should-be-one-of-mr-ap-ji-but-got","errorCode":null,"errorMessage":"metric should be one of 'MR', 'AP', 'JI',but got {metric}.","messagePattern":"metric should be one of 'MR', 'AP', 'JI',but got (.+?)\\.","errorType":"validation","errorClass":"KeyError","httpStatus":null,"severity":"error","filePath":"mmdet/evaluation/metrics/crowdhuman_metric.py","lineNumber":89,"sourceCode":"                 outfile_prefix: Optional[str] = None,\n                 file_client_args: dict = None,\n                 backend_args: dict = None,\n                 collect_device: str = 'cpu',\n                 prefix: Optional[str] = None,\n                 eval_mode: int = 0,\n                 iou_thres: float = 0.5,\n                 compare_matching_method: Optional[str] = None,\n                 mr_ref: str = 'CALTECH_-2',\n                 num_ji_process: int = 10) -> None:\n        super().__init__(collect_device=collect_device, prefix=prefix)\n\n        self.ann_file = ann_file\n        # crowdhuman evaluation metrics\n        self.metrics = metric if isinstance(metric, list) else [metric]\n        allowed_metrics = ['MR', 'AP', 'JI']\n        for metric in self.metrics:\n            if metric not in allowed_metrics:\n                raise KeyError(f\"metric should be one of 'MR', 'AP', 'JI',\"\n                               f'but got {metric}.')\n\n        self.format_only = format_only\n        if self.format_only:\n            assert outfile_prefix is not None, 'outfile_prefix must be not'\n            'None when format_only is True, otherwise the result files will'\n            'be saved to a temp directory which will be cleaned up at the end.'\n        self.outfile_prefix = outfile_prefix\n        self.backend_args = backend_args\n        if file_client_args is not None:\n            raise RuntimeError(\n                'The `file_client_args` is deprecated, '\n                'please use `backend_args` instead, please refer to'\n                'https://github.com/open-mmlab/mmdetection/blob/main/configs/_base_/datasets/coco_detection.py'  # noqa: E501\n            )\n\n        assert eval_mode in [0, 1, 2], \\\n            \"Unknown eval mode. mr_ref should be one of '0', '1', '2'.\"","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/open-mmlab/mmdetection/blob/cfd5d3a985b0249de009b67d04f37263e11cdf3d/mmdet/evaluation/metrics/crowdhuman_metric.py#L71-L107","documentation":"CrowdHumanMetric.__init__ validates the `metric` argument against CrowdHuman's supported metrics: 'MR' (log-average miss rate), 'AP', 'JI' (Jaccard Index). Any other value raises KeyError. This is the CrowdHuman analogue of CocoMetric's metric whitelist and fails at evaluator construction, before any eval runs.","triggerScenarios":"Passing metric='bbox' (a COCO name) to CrowdHumanMetric, or misspelled/lowercase names like 'ap'/'ji '; mixing metric lists from COCO configs into a CrowdHuman config.","commonSituations":"Adapting a COCO detection config to CrowdHuman and leaving metric=['bbox']; copy-paste between evaluator types; assuming mmdet metric names are universal across datasets.","solutions":["Use only 'MR', 'AP', 'JI' or a list of them, e.g. dict(type='CrowdHumanMetric', metric=['MR','AP','JI'])","If you wanted COCO metrics, you are using the wrong metric class — switch to CocoMetric","Double-check the dataset type matches the evaluator type in the config"],"exampleFix":"# before\nval_evaluator = dict(type='CrowdHumanMetric', ann_file=..., metric='bbox')\n# after\nval_evaluator = dict(type='CrowdHumanMetric', ann_file=..., metric=['AP','MR','JI'])","handlingStrategy":"validation","validationCode":"ALLOWED = {'MR', 'AP', 'JI'}\nmetrics = cfg['val_evaluator']['metric']\nmetrics = [metrics] if isinstance(metrics, str) else metrics\nbad = [m for m in metrics if m not in ALLOWED]\nassert not bad, f'Invalid CrowdHumanMetric metric(s): {bad}; allowed: MR/AP/JI'","typeGuard":"def is_valid_crowdhuman_metric(m) -> bool:\n    items = [m] if isinstance(m, str) else m\n    return bool(items) and all(x in {'MR', 'AP', 'JI'} for x in items)","tryCatchPattern":null,"preventionTips":["Per-dataset metric whitelists differ — check the metric class, not a global list","Pair dataset_type with the matching evaluator type in configs","Unit-test evaluator construction from config dicts"],"tags":["mmdetection","crowdhuman","metric-validation","config-error"],"backgroundTag":"invalid-argument-value","analyzedSha":"cfd5d3a985b0249de009b67d04f37263e11cdf3d","analyzedAt":"2026-08-27T20:54:20.183Z","schemaVersion":2},"datasetVersion":"2026-08-28T00:17:15.603Z"}