roboflow/supervision · error · NotImplementedError

Evaluating predictions with segmentation is not supported.

Error message

Evaluating predictions with segmentation is not supported.

What it means

EvaluationDataset.load_predictions() only implements box-detection results. This NotImplementedError fires when the first prediction dict contains a 'segmentation' key — the COCO segmentation result format (RLE or polygon masks). The library's COCO backend does not evaluate segmentation predictions in this path; mask-based evaluation must instead go through MeanAveragePrecision with metric_target=MetricTarget.MASKS and sv.Detections carrying .mask.

Source

Thrown at src/supervision/metrics/mean_average_precision.py:542

        if len(predictions) == 0:
            predictions_dataset.dataset["annotations"] = []
            return predictions_dataset

        ids = [pred["image_id"] for pred in predictions]

        # Make sure the image ids from predictions exist in the current dataset.
        # A plain ``assert`` would be stripped under ``python -O``, so validate
        # this public-input contract with an explicit exception instead.
        if not set(ids) <= set(self.get_image_ids()):
            raise ValueError("Results do not correspond to current coco set")

        # Check if the predictions contain any unsupported keys
        if "caption" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with caption is not supported."
            )
        elif "segmentation" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with segmentation is not supported."
            )
        elif "keypoints" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with keypoints is not supported."
            )

        elif "bbox" in predictions[0] and not predictions[0]["bbox"] == []:
            predictions_dataset.dataset["categories"] = copy.deepcopy(
                self.dataset["categories"]
            )

            # Prepare fields for every prediction of the given image
            for idx, pred in enumerate(predictions):
                x, y, w, h = pred["bbox"]
                x1, x2, y1, y2 = [x, x + w, y, y + h]

                # Make segmentation from bounding box coordinates

View on GitHub (pinned to 7f254d9784)

Solutions

For mask evaluation, use MeanAveragePrecision(metric_target=MetricTarget.MASKS) with Detections whose .mask is populated (segmentation connectors like from_ultralytics on YOLO-seg populate it)
Strip 'segmentation' keys only if you truly want box-only evaluation of those results
Use pycocotools' COCOeval with iouType='segm' for official RLE-based COCO segmentation scoring
Route result files by task before evaluation: check keys of the first entry

Example fix

# before
seg_results = [{'image_id': 1, 'category_id': 2,
                'segmentation': [[...polygon...]], 'score': 0.9}]
coco_det = coco_gt.load_predictions(seg_results)

# after (mask mAP via the Detections API)
map_ = sv.MeanAveragePrecision(metric_target=sv.MetricTarget.MASKS)
map_.update(sv.Detections(xyxy=boxes, mask=masks, class_id=ids,
                          confidence=confs),
            sv.Detections(xyxy=gt_boxes, mask=gt_masks, class_id=gt_ids))
result = map_.compute()

Defensive patterns

Strategy: validation

Validate before calling

if predictions and 'segmentation' in predictions[0]:
    raise TypeError('use MeanAveragePrecision with MetricTarget.MASKS for masks, '
                    'or pycocotools COCOeval segm; this API is box-only')

Type guard

def unsupported_keys(preds: list) -> set:
    """Return COCO task keys this evaluator cannot handle."""
    return {'caption', 'segmentation', 'keypoints'} & (set(preds[0]) if preds else set())

Try / catch

try:
    dataset.load_predictions(results)
except NotImplementedError as e:
    if 'segmentation' in str(e):
        results = [{k: v for k, v in r.items() if k != 'segmentation'}
                   for r in results]  # only if box-only eval is intended
        coco_det = dataset.load_predictions(results)
    else:
        raise

Prevention

For mask mAP use MetricTarget.MASKS with Detections.mask
Use pycocotools COCOeval(iouType='segm') for official RLE scoring
Split multi-task result files before evaluation

When it happens

Trigger: Loading COCO instance-segmentation results files (with 'segmentation' RLE/polygons) into load_predictions; converting a panoptic/segmentation model's output to COCO dict format and feeding it here; mixing detection and segmentation entries where the first entry has a segmentation key.

Common situations: Evaluating YOLO-seg / Mask R-CNN outputs saved in official COCO results format; assuming pycocotools-style segm evaluation is available; reusing detection evaluation scripts unchanged for segmentation models.

Related errors

AI-assisted analysis of roboflow/supervision@7f254d9784 (2026-08-15). Data as JSON: /api/errors/5b36e27d638a6fae. Report an issue: GitHub.