{"record":{"id":"b1349beca481e610","repo":"apache/iceberg","slug":"deleting-a-duplicate-path-from-manifest","errorCode":null,"errorMessage":"Deleting a duplicate path from manifest {}: {}","messagePattern":"Deleting a duplicate path from manifest (.+?): (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"core/src/main/java/org/apache/iceberg/ManifestFilterManager.java","lineNumber":542,"sourceCode":"                          && entry.isLive()\n                          && entry.dataSequenceNumber() > 0\n                          && entry.dataSequenceNumber() < minSequenceNumber);\n              if (markedForDelete || evaluator.rowsMightMatch(file)) {\n                boolean allRowsMatch = markedForDelete || evaluator.rowsMustMatch(file);\n                ValidationException.check(\n                    allRowsMatch\n                        || isDelete, // ignore delete files where some records may not match\n                    // the expression\n                    \"Cannot delete file where some, but not all, rows match filter %s: %s\",\n                    this.deleteExpression,\n                    file.location());\n\n                if (allRowsMatch) {\n                  writer.delete(entry);\n                  F fileCopy = file.copyWithoutStats();\n\n                  if (deletedFiles.contains(file)) {\n                    LOG.warn(\n                        \"Deleting a duplicate path from manifest {}: {}\",\n                        manifest.path(),\n                        file.location());\n                    duplicateDeleteCount.incrementAndGet();\n                  } else {\n                    // only add the file to deletes if it is a new delete\n                    // this keeps the snapshot summary accurate for non-duplicate data\n                    deletedFiles.add(fileCopy);\n                  }\n                } else {\n                  writer.existing(entry);\n                }\n\n              } else {\n                writer.existing(entry);\n              }\n            });\n      } catch (IOException e) {","sourceCodeStart":524,"sourceCodeEnd":560,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L524-L560","documentation":"ManifestFilterManager.filterManifestWithDeletedFiles warns when the same file path is deleted more than once within one manifest during filtering — i.e. the requested deletedFiles set contains a duplicate path already deleted by an earlier entry in this manifest. The code counts it as duplicateDeleteCount, deletes the entry once, and proceeds; it signals duplicate input rather than data loss.","triggerScenarios":"A delete operation (DeleteFiles, overwrite with partial progress, custom row/file delete API) passes a set/list of files to delete that contains the same path twice — e.g. files collected via a non-deduplicating collection across retries or overlapping scans.","commonSituations":"Application code gathering files-to-delete with a List instead of a Set; replaying a delete request after a partial failure; combining results of multiple scans that overlap.","solutions":["Deduplicate file paths before passing them to delete (collect into a Set by path/location).","Ensure delete-file collection is done from a single consistent snapshot rather than merged overlapping scans.","Check for retry logic that re-invokes deleteFiles with the same file set after a timeout; make the operation idempotent at the call site."],"exampleFix":"// before\nList<DataFile> filesToDelete = scanFiles(...);\ntable.newDelete().deleteFiles(filesToDelete).commit();\n// after\nSet<String> seen = new HashSet<>();\nList<DataFile> filesToDelete = scanFiles(...).stream()\n    .filter(f -> seen.add(f.location()))\n    .collect(Collectors.toList());\ntable.newDelete().deleteFiles(filesToDelete).commit();","handlingStrategy":"validation","validationCode":"Set<String> locations = files.stream().map(DataFile::location)\n    .collect(Collectors.toCollection(LinkedHashSet::new));\nif (locations.size() < files.size()) {\n  throw new IllegalArgumentException(\"Duplicate file paths passed to delete\");\n}","typeGuard":"static boolean hasNoDuplicatePaths(List<DataFile> files) {\n  return files.stream().map(DataFile::location).distinct().count() == files.size();\n}","tryCatchPattern":null,"preventionTips":["Collect files-to-delete into path-deduplicating collections (Set keyed by location).","Gather delete targets from a single snapshot/scan, not merged overlapping scans.","Make delete jobs idempotent so retries do not re-submit already deleted files.","Treat this WARN as a signal of a caller-side deduplication bug."],"tags":["manifests","duplicate-data","deletion"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}