roboflow/supervision · error · ValueError

Image paths {duplicates} are not unique across datasets.

Error message

Image paths {duplicates} are not unique across datasets.

What it means

Raised by DetectionDataset.merge() when the same image path appears in more than one (or duplicated within) input datasets. After merge, annotations are combined into a single dict keyed by image path — duplicate keys would overwrite each other and silently lose annotations, so supervision detects duplicates (via find_duplicates) and raises.

Source

Thrown at src/supervision/dataset/core.py:349

        all_in_memory = all([is_in_memory(dataset) for dataset in dataset_list])
        all_lazy = all([is_lazy(dataset) for dataset in dataset_list])
        if not all_in_memory and not all_lazy:
            raise ValueError(
                "Merging lazy and in-memory DetectionDatasets is not supported."
            )

        images_in_memory = {}
        for dataset in dataset_list:
            images_in_memory.update(dataset._images_in_memory)

        image_paths = list(
            chain.from_iterable(dataset.image_paths for dataset in dataset_list)
        )
        image_paths_unique = list(dict.fromkeys(image_paths))
        if len(image_paths) != len(image_paths_unique):
            duplicates = find_duplicates(image_paths)
            raise ValueError(
                f"Image paths {duplicates} are not unique across datasets."
            )
        image_paths = image_paths_unique

        classes = merge_class_lists(
            class_lists=[dataset.classes for dataset in dataset_list]
        )

        annotations = {}
        for dataset in dataset_list:
            annotations.update(dataset.annotations)
        for dataset in dataset_list:
            class_index_mapping = build_class_index_mapping(
                source_classes=dataset.classes, target_classes=classes
            )
            for image_path in dataset.image_paths:
                annotations[image_path] = map_detections_class_id(
                    source_to_target_mapping=class_index_mapping,

View on GitHub (pinned to 7f254d9784)

Solutions

1. Deduplicate image paths across datasets before merging: drop duplicates from all but one dataset.
2. Rebuild the datasets with disjoint image lists (proper train/val split).
3. If the same path was intentionally annotated twice, keep them as separate datasets rather than merging.

Example fix

// before
merged = DetectionDataset.merge([ds_a, ds_b])  # overlapping paths

// after
seen = set(ds_a.image_paths)
ds_b = DetectionDataset(
    classes=ds_b.classes,
    images=[p for p in ds_b.image_paths if p not in seen],
    annotations={p: a for p, a in ds_b.annotations.items() if p not in seen},
)
merged = DetectionDataset.merge([ds_a, ds_b])

all_paths = [p for ds in dataset_list for p in ds.image_paths]
dups = {p for p in all_paths if all_paths.count(p) > 1}
assert not dups, f"Duplicate image paths across datasets: {dups}"
merged = DetectionDataset.merge(dataset_list)

def no_shared_paths(datasets: list[DetectionDataset]) -> bool:
    seen = set()
    for ds in datasets:
        if set(ds.image_paths) & seen:
            return False
        seen |= set(ds.image_paths)
    return True

When it happens

Trigger: Calling DetectionDataset.merge([ds_train, ds_val]) where ds_train.image_paths and ds_val.image_paths share entries — e.g. both built from overlapping globs, or the same dataset merged with itself.

Common situations: Merging train/validation splits that overlap; combining a dataset with a superset of itself; deduplication forgotten when building multiple datasets from one directory tree.

Related errors

• Class {class_name} not found in target classes. source_class
• All KeyPoints must have the same coordinate depth per skelet
• All or none of the '{name}' fields must be None
• Detections must have class_id attribute.
• Detections class_id must be a subset of source_to_target_map

AI-assisted analysis of roboflow/supervision@7f254d9784 (2026-08-15). Data as JSON: /api/errors/9809a36f73a6e15d. Report an issue: GitHub.