{"record":{"id":"9809a36f73a6e15d","repo":"roboflow/supervision","slug":"image-paths-duplicates-are-not-unique-across-dat","errorCode":null,"errorMessage":"Image paths {duplicates} are not unique across datasets.","messagePattern":"Image paths (.+?) are not unique across datasets\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/supervision/dataset/core.py","lineNumber":349,"sourceCode":"\n        all_in_memory = all([is_in_memory(dataset) for dataset in dataset_list])\n        all_lazy = all([is_lazy(dataset) for dataset in dataset_list])\n        if not all_in_memory and not all_lazy:\n            raise ValueError(\n                \"Merging lazy and in-memory DetectionDatasets is not supported.\"\n            )\n\n        images_in_memory = {}\n        for dataset in dataset_list:\n            images_in_memory.update(dataset._images_in_memory)\n\n        image_paths = list(\n            chain.from_iterable(dataset.image_paths for dataset in dataset_list)\n        )\n        image_paths_unique = list(dict.fromkeys(image_paths))\n        if len(image_paths) != len(image_paths_unique):\n            duplicates = find_duplicates(image_paths)\n            raise ValueError(\n                f\"Image paths {duplicates} are not unique across datasets.\"\n            )\n        image_paths = image_paths_unique\n\n        classes = merge_class_lists(\n            class_lists=[dataset.classes for dataset in dataset_list]\n        )\n\n        annotations = {}\n        for dataset in dataset_list:\n            annotations.update(dataset.annotations)\n        for dataset in dataset_list:\n            class_index_mapping = build_class_index_mapping(\n                source_classes=dataset.classes, target_classes=classes\n            )\n            for image_path in dataset.image_paths:\n                annotations[image_path] = map_detections_class_id(\n                    source_to_target_mapping=class_index_mapping,","sourceCodeStart":331,"sourceCodeEnd":367,"githubUrl":"https://github.com/roboflow/supervision/blob/7f254d9784d4c37e0f03cd89ddee164c8db099c0/src/supervision/dataset/core.py#L331-L367","documentation":"Raised by DetectionDataset.merge() when the same image path appears in more than one (or duplicated within) input datasets. After merge, annotations are combined into a single dict keyed by image path — duplicate keys would overwrite each other and silently lose annotations, so supervision detects duplicates (via find_duplicates) and raises.","triggerScenarios":"Calling DetectionDataset.merge([ds_train, ds_val]) where ds_train.image_paths and ds_val.image_paths share entries — e.g. both built from overlapping globs, or the same dataset merged with itself.","commonSituations":"Merging train/validation splits that overlap; combining a dataset with a superset of itself; deduplication forgotten when building multiple datasets from one directory tree.","solutions":["Deduplicate image paths across datasets before merging: drop duplicates from all but one dataset.","Rebuild the datasets with disjoint image lists (proper train/val split).","If the same path was intentionally annotated twice, keep them as separate datasets rather than merging."],"exampleFix":"// before\nmerged = DetectionDataset.merge([ds_a, ds_b])  # overlapping paths\n\n// after\nseen = set(ds_a.image_paths)\nds_b = DetectionDataset(\n    classes=ds_b.classes,\n    images=[p for p in ds_b.image_paths if p not in seen],\n    annotations={p: a for p, a in ds_b.annotations.items() if p not in seen},\n)\nmerged = DetectionDataset.merge([ds_a, ds_b])","handlingStrategy":"validation","validationCode":"all_paths = [p for ds in dataset_list for p in ds.image_paths]\ndups = {p for p in all_paths if all_paths.count(p) > 1}\nassert not dups, f\"Duplicate image paths across datasets: {dups}\"\nmerged = DetectionDataset.merge(dataset_list)","typeGuard":"def no_shared_paths(datasets: list[DetectionDataset]) -> bool:\n    seen = set()\n    for ds in datasets:\n        if set(ds.image_paths) & seen:\n            return False\n        seen |= set(ds.image_paths)\n    return True","tryCatchPattern":null,"preventionTips":["Ensure train/val/test splits are disjoint before merging anything.","Deduplicate image paths across datasets before calling merge().","Never merge a dataset with itself or a subset of itself."],"tags":["dataset","merge","duplicates","validation"],"backgroundTag":null,"analyzedSha":"7f254d9784d4c37e0f03cd89ddee164c8db099c0","analyzedAt":"2026-08-15T05:13:01.950Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}