roboflow/supervision · error · NotImplementedError

Evaluating predictions with caption is not supported.

Error message

Evaluating predictions with caption is not supported.

What it means

EvaluationDataset.load_predictions() supports only box-style detection results (image_id, category_id, bbox, score). This NotImplementedError fires when the first prediction dict contains a 'caption' key — the caption/detection-captioning result format. The COCO evaluation backend in this library does not implement caption evaluation, so the input is rejected rather than half-evaluated.

Source

Thrown at src/supervision/metrics/mean_average_precision.py:538

        if not isinstance(predictions, list):
            raise ValueError("results must be a list")

        # Handle empty predictions
        if len(predictions) == 0:
            predictions_dataset.dataset["annotations"] = []
            return predictions_dataset

        ids = [pred["image_id"] for pred in predictions]

        # Make sure the image ids from predictions exist in the current dataset.
        # A plain ``assert`` would be stripped under ``python -O``, so validate
        # this public-input contract with an explicit exception instead.
        if not set(ids) <= set(self.get_image_ids()):
            raise ValueError("Results do not correspond to current coco set")

        # Check if the predictions contain any unsupported keys
        if "caption" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with caption is not supported."
            )
        elif "segmentation" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with segmentation is not supported."
            )
        elif "keypoints" in predictions[0]:
            raise NotImplementedError(
                "Evaluating predictions with keypoints is not supported."
            )

        elif "bbox" in predictions[0] and not predictions[0]["bbox"] == []:
            predictions_dataset.dataset["categories"] = copy.deepcopy(
                self.dataset["categories"]
            )

            # Prepare fields for every prediction of the given image
            for idx, pred in enumerate(predictions):

View on GitHub (pinned to 7f254d9784)

Solutions

1. Use box detection results only: dicts with image_id, category_id, bbox, score
2. Split multi-task result files and pass only the detection entries
3. For caption evaluation use pycocoevalcap or the official COCO caption evaluation tools, not supervision
4. Check predictions[0] keys before calling load_predictions and route each format to its proper evaluator

Example fix

# before
caption_results = [{'image_id': 1, 'caption': 'a dog on a beach'}]
coco_det = coco_gt.load_predictions(caption_results)

# after
box_results = [{'image_id': 1, 'category_id': 18,
                'bbox': [x, y, w, h], 'score': 0.92}]
coco_det = coco_gt.load_predictions(box_results)

if predictions and 'caption' in predictions[0]:
    raise TypeError('caption results are not supported; use a caption evaluator')

def is_box_result_list(preds: list) -> bool:
    """True when first result has bbox-style keys and no unsupported keys."""
    if not preds:
        return True
    return not ({'caption', 'segmentation', 'keypoints'} & set(preds[0]))

try:
    dataset.load_predictions(results)
except NotImplementedError:
    log.error('non-box COCO results; routing to task-specific evaluator')
    raise

When it happens

Trigger: Feeding COCO 'caption' task results (e.g. from image-captioning models, dense-captioning outputs where each result has 'caption' plus image_id) into load_predictions; submitting results from pycocoevalcap-style files to supervision's evaluator; mixing task outputs in one results file where the first entry is a caption result.

Common situations: Running multi-task models (captioning + detection) and pointing the evaluator at the wrong results file; converting between COCO task formats; assuming supervision's MeanAveragePrecision handles all COCO result types like the official pycocotools suite.

Related errors

• Evaluating predictions with segmentation is not supported.
• Evaluating predictions with keypoints is not supported.
• results must be a list
• Results do not correspond to current coco set
• coco_targets must be provided

AI-assisted analysis of roboflow/supervision@7f254d9784 (2026-08-15). Data as JSON: /api/errors/65ccbb8492ea135e. Report an issue: GitHub.