roboflow/supervision · error · NotImplementedError

Evaluating predictions with keypoints is not supported.

Error message

Evaluating predictions with keypoints is not supported.

What it means

EvaluationDataset.load_predictions() implements box-detection evaluation only. This NotImplementedError fires when the first prediction dict contains a 'keypoints' key — the COCO keypoint-result format (person keypoints, pose models). Keypoint/pose evaluation is out of scope for this backend, so the input is rejected explicitly rather than evaluated incorrectly.

Source

Thrown at src/supervision/metrics/mean_average_precision.py:546

        ids = [pred["image_id"] for pred in predictions]

        # Make sure the image ids from predictions exist in the current dataset.
        # A plain ``assert`` would be stripped under ``python -O``, so validate
        # this public-input contract with an explicit exception instead.
        if not set(ids) <= set(self.get_image_ids()):
            raise ValueError("Results do not correspond to current coco set")

        # Check if the predictions contain any unsupported keys
        if "caption" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with caption is not supported."
            )
        elif "segmentation" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with segmentation is not supported."
            )
        elif "keypoints" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with keypoints is not supported."
            )

        elif "bbox" in predictions[0] and not predictions[0]["bbox"] == []:
            predictions_dataset.dataset["categories"] = copy.deepcopy(
                self.dataset["categories"]
            )

            # Prepare fields for every prediction of the given image
            for idx, pred in enumerate(predictions):
                x, y, w, h = pred["bbox"]
                x1, x2, y1, y2 = [x, x + w, y, y + h]

                # Make segmentation from bounding box coordinates
                if "segmentation" not in pred:
                    pred["segmentation"] = [[x1, y1, x1, y2, x2, y2, x2, y1]]
                # Use provided area if available
                if "area" not in pred:

View on GitHub (pinned to 7f254d9784)

Solutions

1. Use box results (image_id, category_id, bbox, score) for this API
2. For pose evaluation use pycocotools COCOeval with iouType='keypoints' or a pose-specific library
3. Filter mixed result files by key before evaluation
4. For supervision-native keypoint workflows, use supervision.key_points classes rather than the COCO mAP path

Example fix

# before
pose_results = [{'image_id': 1, 'category_id': 1,
                 'keypoints': [...], 'score': 0.87}]
coco_det = coco_gt.load_predictions(pose_results)

# after
box_results = [{'image_id': 1, 'category_id': 1,
                'bbox': [x, y, w, h], 'score': 0.87}]
coco_det = coco_gt.load_predictions(box_results)

if predictions and 'keypoints' in predictions[0]:
    raise TypeError('keypoint results unsupported; use pycocotools keypoint eval')

def is_box_only_results(preds: list) -> bool:
    """True when no unsupported task keys appear in the first result."""
    if not preds:
        return True
    return not ({'caption', 'segmentation', 'keypoints'} & set(preds[0]))

try:
    dataset.load_predictions(results)
except NotImplementedError as e:
    if 'keypoints' in str(e):
        raise RuntimeError('route pose results to COCOeval(iouType="keypoints")') from e
    raise

When it happens

Trigger: Feeding COCO pose results (dicts with image_id, category_id, keypoints, score) into load_predictions; evaluating pose-model exports (e.g. from a keypoint detector) saved in official COCO keypoint format; mixed-task results files whose first record is a keypoint result.

Common situations: Running pose estimation benchmarks and reaching for supervision's mAP; reusing detection eval scripts on pose outputs; converting between COCO task JSONs without dropping task-specific keys.

Related errors

• Evaluating predictions with caption is not supported.
• Evaluating predictions with segmentation is not supported.
• results must be a list
• Results do not correspond to current coco set
• coco_targets must be provided

AI-assisted analysis of roboflow/supervision@7f254d9784 (2026-08-15). Data as JSON: /api/errors/26b2e57c7cb4d314. Report an issue: GitHub.