What causes "Please run accumulate() first" in open-mmlab/mmdetection?

Ensure the canonical order: coco_eval.evaluate(); coco_eval.accumulate(); coco_eval.summarize()

open-mmlab/mmdetection · error · Exception

Please run accumulate() first

Error message

Please run accumulate() first

What it means

COCOevalMp.summarize() requires that accumulate() has been run first, because summarize reads self.eval['counts'] etc. populated by accumulate. If self.eval is falsy (not yet computed, or empty after evaluating zero results), it raises Exception('Please run accumulate() first'). This mirrors upstream pycocotools COCOeval.summarize behavior in the multiprocessing variant.

Source

Thrown at mmdet/datasets/api_wrappers/cocoeval_mp.py:290

            stats = np.array(stats)
            return stats

        def _summarizeKps():
            stats = np.zeros((10, ))
            stats[0] = _summarize(1, maxDets=20)
            stats[1] = _summarize(1, maxDets=20, iouThr=.5)
            stats[2] = _summarize(1, maxDets=20, iouThr=.75)
            stats[3] = _summarize(1, maxDets=20, areaRng='medium')
            stats[4] = _summarize(1, maxDets=20, areaRng='large')
            stats[5] = _summarize(0, maxDets=20)
            stats[6] = _summarize(0, maxDets=20, iouThr=.5)
            stats[7] = _summarize(0, maxDets=20, iouThr=.75)
            stats[8] = _summarize(0, maxDets=20, areaRng='medium')
            stats[9] = _summarize(0, maxDets=20, areaRng='large')
            return stats

        if not self.eval:
            raise Exception('Please run accumulate() first')
        iouType = self.params.iouType
        if iouType == 'segm' or iouType == 'bbox':
            summarize = _summarizeDets
        elif iouType == 'keypoints':
            summarize = _summarizeKps
        self.stats = summarize()

View on GitHub (pinned to cfd5d3a985)

Solutions

Ensure the canonical order: coco_eval.evaluate(); coco_eval.accumulate(); coco_eval.summarize()
If it still fires with empty predictions, check that the model actually detects anything (log number of dets) and that ann labels/CATEGORY ids match
Catch the exception when running evaluations where zero predictions are legitimate, and report metrics as N/A

Example fix

# before
coco_eval.evaluate()
stats = coco_eval.summarize()  # Exception: Please run accumulate() first
# after
coco_eval.evaluate()
coco_eval.accumulate()
stats = coco_eval.summarize()

Defensive patterns

Strategy: validation

Validate before calling

coco_eval.evaluate()
if not coco_eval.eval.get('counts', None) and not coco_eval.evalImgs:
    raise RuntimeError('evaluate() produced no results; check predictions/annotations')
coco_eval.accumulate()
coco_eval.summarize()

Type guard

def is_ready_to_summarize(coco_eval) -> bool:
    return bool(getattr(coco_eval, 'eval', None)) and bool(getattr(coco_eval, 'params', None))

Try / catch

try:
    stats = coco_eval.summarize()
except Exception as e:
    if 'accumulate' in str(e):
        coco_eval.accumulate()
        stats = coco_eval.summarize()
    else:
        raise

Prevention

Always call evaluate -> accumulate -> summarize in that order, ideally wrapped in one helper
Warn when total detections == 0 after evaluate, since empty evalImgs leads here even with correct ordering
Unit-test eval helpers against a tiny known-good prediction set

When it happens

Trigger: Calling summarize() directly after evaluate() but before accumulate(); or calling summarize when evaluate() produced no detections (self.eval empty), which yields the same guard trip even if accumulate was nominally called on empty results.

Common situations: Custom eval loops that reorder COCOeval steps; evaluating a checkpoint that predicts nothing (untrained model, wrong classes) so evalImgs is empty; multiprocessing wrapper (cocoeval_mp) used with a subset that skipped accumulation.

Related errors

AI-assisted analysis of open-mmlab/mmdetection@cfd5d3a985 (2026-08-27). Data as JSON: /api/errors/9b1b1d497deda070. Report an issue: GitHub.