facebookresearch/detectron2 · error · TypeError

Unknown segmentation type {type(segmentation)}!

Error message

Unknown segmentation type {type(segmentation)}!

What it means

When converting annotations to COCO dict format and computing area from segmentation, only list (polygon) and dict (RLE) segmentations are supported. Any other type (e.g. a numpy array mask, bytes, torch.Tensor, None, str) raises TypeError.

Source

Thrown at detectron2/data/datasets/coco.py:387

                bbox = bbox.tolist()
            if len(bbox) not in [4, 5]:
                raise ValueError(f"bbox has to has length 4 or 5. Got {bbox}.")
            from_bbox_mode = annotation["bbox_mode"]
            to_bbox_mode = BoxMode.XYWH_ABS if len(bbox) == 4 else BoxMode.XYWHA_ABS
            bbox = BoxMode.convert(bbox, from_bbox_mode, to_bbox_mode)

            # COCO requirement: instance area
            if "segmentation" in annotation:
                # Computing areas for instances by counting the pixels
                segmentation = annotation["segmentation"]
                # TODO: check segmentation type: RLE, BinaryMask or Polygon
                if isinstance(segmentation, list):
                    polygons = PolygonMasks([segmentation])
                    area = polygons.area()[0].item()
                elif isinstance(segmentation, dict):  # RLE
                    area = mask_util.area(segmentation).item()
                else:
                    raise TypeError(f"Unknown segmentation type {type(segmentation)}!")
            else:
                # Computing areas using bounding boxes
                if to_bbox_mode == BoxMode.XYWH_ABS:
                    bbox_xy = BoxMode.convert(bbox, to_bbox_mode, BoxMode.XYXY_ABS)
                    area = Boxes([bbox_xy]).area()[0].item()
                else:
                    area = RotatedBoxes([bbox]).area()[0].item()

            if "keypoints" in annotation:
                keypoints = annotation["keypoints"]  # list[int]
                for idx, v in enumerate(keypoints):
                    if idx % 3 != 2:
                        # COCO's segmentation coordinates are floating points in [0, H or W],
                        # but keypoint coordinates are integers in [0, H-1 or W-1]
                        # For COCO format consistency we substract 0.5
                        # https://github.com/facebookresearch/detectron2/pull/175#issuecomment-551202163
                        keypoints[idx] = v - 0.5
                if "num_keypoints" in annotation:

View on GitHub (pinned to a2f4a8771a)

Solutions

1. Convert masks to RLE dict with pycocotools: mask_util.encode(np.asfortranarray(mask.astype(np.uint8))) and keep the dict
2. Or convert mask to polygons via cv2.findContours and store as list[list[float]]
3. Pass segmentation=None/omit when only boxes are available

Example fix

# before
ann['segmentation'] = mask  # np.ndarray HxW
# after
from pycocotools import mask as mask_util
rle = mask_util.encode(np.asfortranarray(mask.astype(np.uint8)))
rle['counts'] = rle['counts'].decode('utf-8')
ann['segmentation'] = rle

seg = annotation.get('segmentation')
assert seg is None or isinstance(seg, (list, dict)), \
    f'segmentation must be polygon list or RLE dict, got {type(seg)}'

def is_coco_segmentation(s) -> bool:
    return s is None or isinstance(s, (list, dict))

When it happens

Trigger: Calling convert_to_coco_dict with annotation['segmentation'] as a binary mask ndarray or torch tensor; encoding masks yourself as encoded-RLE bytes instead of the dict form {'size':..., 'counts':...}.

Common situations: See trigger scenarios.

Related errors

• Unsupported type for argument `serailzie`: {serialize}
• bbox has to be 1-dimensional. Got shape={bbox.shape}.
• bbox has to has length 4 or 5. Got {bbox}.
• Cannot match one checkpoint key to multiple keys in the mode
• Class with @configurable must have a 'from_config' classmeth

AI-assisted analysis of facebookresearch/detectron2@a2f4a8771a (2026-08-27). Data as JSON: /api/errors/7865ace7ce659ea8. Report an issue: GitHub.