{"record":{"id":"65ccbb8492ea135e","repo":"roboflow/supervision","slug":"evaluating-predictions-with-caption-is-not-support","errorCode":null,"errorMessage":"Evaluating predictions with caption is not supported.","messagePattern":"Evaluating predictions with caption is not supported\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/supervision/metrics/mean_average_precision.py","lineNumber":538,"sourceCode":"        if not isinstance(predictions, list):\n            raise ValueError(\"results must be a list\")\n\n        # Handle empty predictions\n        if len(predictions) == 0:\n            predictions_dataset.dataset[\"annotations\"] = []\n            return predictions_dataset\n\n        ids = [pred[\"image_id\"] for pred in predictions]\n\n        # Make sure the image ids from predictions exist in the current dataset.\n        # A plain ``assert`` would be stripped under ``python -O``, so validate\n        # this public-input contract with an explicit exception instead.\n        if not set(ids) <= set(self.get_image_ids()):\n            raise ValueError(\"Results do not correspond to current coco set\")\n\n        # Check if the predictions contain any unsupported keys\n        if \"caption\" in predictions[0]:\n            raise NotImplementedError(\n                \"Evaluating predictions with caption is not supported.\"\n            )\n        elif \"segmentation\" in predictions[0]:\n            raise NotImplementedError(\n                \"Evaluating predictions with segmentation is not supported.\"\n            )\n        elif \"keypoints\" in predictions[0]:\n            raise NotImplementedError(\n                \"Evaluating predictions with keypoints is not supported.\"\n            )\n\n        elif \"bbox\" in predictions[0] and not predictions[0][\"bbox\"] == []:\n            predictions_dataset.dataset[\"categories\"] = copy.deepcopy(\n                self.dataset[\"categories\"]\n            )\n\n            # Prepare fields for every prediction of the given image\n            for idx, pred in enumerate(predictions):","sourceCodeStart":520,"sourceCodeEnd":556,"githubUrl":"https://github.com/roboflow/supervision/blob/7f254d9784d4c37e0f03cd89ddee164c8db099c0/src/supervision/metrics/mean_average_precision.py#L520-L556","documentation":"EvaluationDataset.load_predictions() supports only box-style detection results (image_id, category_id, bbox, score). This NotImplementedError fires when the first prediction dict contains a 'caption' key — the caption/detection-captioning result format. The COCO evaluation backend in this library does not implement caption evaluation, so the input is rejected rather than half-evaluated.","triggerScenarios":"Feeding COCO 'caption' task results (e.g. from image-captioning models, dense-captioning outputs where each result has 'caption' plus image_id) into load_predictions; submitting results from pycocoevalcap-style files to supervision's evaluator; mixing task outputs in one results file where the first entry is a caption result.","commonSituations":"Running multi-task models (captioning + detection) and pointing the evaluator at the wrong results file; converting between COCO task formats; assuming supervision's MeanAveragePrecision handles all COCO result types like the official pycocotools suite.","solutions":["Use box detection results only: dicts with image_id, category_id, bbox, score","Split multi-task result files and pass only the detection entries","For caption evaluation use pycocoevalcap or the official COCO caption evaluation tools, not supervision","Check predictions[0] keys before calling load_predictions and route each format to its proper evaluator"],"exampleFix":"# before\ncaption_results = [{'image_id': 1, 'caption': 'a dog on a beach'}]\ncoco_det = coco_gt.load_predictions(caption_results)\n\n# after\nbox_results = [{'image_id': 1, 'category_id': 18,\n                'bbox': [x, y, w, h], 'score': 0.92}]\ncoco_det = coco_gt.load_predictions(box_results)","handlingStrategy":"validation","validationCode":"if predictions and 'caption' in predictions[0]:\n    raise TypeError('caption results are not supported; use a caption evaluator')","typeGuard":"def is_box_result_list(preds: list) -> bool:\n    \"\"\"True when first result has bbox-style keys and no unsupported keys.\"\"\"\n    if not preds:\n        return True\n    return not ({'caption', 'segmentation', 'keypoints'} & set(preds[0]))","tryCatchPattern":"try:\n    dataset.load_predictions(results)\nexcept NotImplementedError:\n    log.error('non-box COCO results; routing to task-specific evaluator')\n    raise","preventionTips":["Keep one results file per COCO task","Route files by inspecting the first record's keys","Use pycocoevalcap for captions"],"tags":["metrics","mean-average-precision","coco","not-implemented","captions"],"backgroundTag":null,"analyzedSha":"7f254d9784d4c37e0f03cd89ddee164c8db099c0","analyzedAt":"2026-08-15T05:13:01.950Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}