apache/iceberg · warning

Deleting a duplicate path from manifest

Error message

Deleting a duplicate path from manifest {}: {}

What it means

ManifestFilterManager.filterManifestWithDeletedFiles warns when the same file path is deleted more than once within one manifest during filtering — i.e. the requested deletedFiles set contains a duplicate path already deleted by an earlier entry in this manifest. The code counts it as duplicateDeleteCount, deletes the entry once, and proceeds; it signals duplicate input rather than data loss.

Solutions

1. Deduplicate file paths before passing them to delete (collect into a Set by path/location).
2. Ensure delete-file collection is done from a single consistent snapshot rather than merged overlapping scans.
3. Check for retry logic that re-invokes deleteFiles with the same file set after a timeout; make the operation idempotent at the call site.

Example fix

// before
List<DataFile> filesToDelete = scanFiles(...);
table.newDelete().deleteFiles(filesToDelete).commit();
// after
Set<String> seen = new HashSet<>();
List<DataFile> filesToDelete = scanFiles(...).stream()
    .filter(f -> seen.add(f.location()))
    .collect(Collectors.toList());
table.newDelete().deleteFiles(filesToDelete).commit();

Set<String> locations = files.stream().map(DataFile::location)
    .collect(Collectors.toCollection(LinkedHashSet::new));
if (locations.size() < files.size()) {
  throw new IllegalArgumentException("Duplicate file paths passed to delete");
}

static boolean hasNoDuplicatePaths(List<DataFile> files) {
  return files.stream().map(DataFile::location).distinct().count() == files.size();
}

When it happens

Trigger: A delete operation (DeleteFiles, overwrite with partial progress, custom row/file delete API) passes a set/list of files to delete that contains the same path twice — e.g. files collected via a non-deduplicating collection across retries or overlapping scans.

Common situations: Application code gathering files-to-delete with a List instead of a Set; replaying a delete request after a partial failure; combining results of multiple scans that overlap.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Bulk deletion failed
• Cannot rewrite manifests in a
• Delete failed for file
• Failed on snapshot while reading manifest list
• Failed to get added files: this may cause orphaned data…

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b1349beca481e610. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/ManifestFilterManager.java:542

                          && entry.isLive()
                          && entry.dataSequenceNumber() > 0
                          && entry.dataSequenceNumber() < minSequenceNumber);
              if (markedForDelete || evaluator.rowsMightMatch(file)) {
                boolean allRowsMatch = markedForDelete || evaluator.rowsMustMatch(file);
                ValidationException.check(
                    allRowsMatch
                        || isDelete, // ignore delete files where some records may not match
                    // the expression
                    "Cannot delete file where some, but not all, rows match filter %s: %s",
                    this.deleteExpression,
                    file.location());

                if (allRowsMatch) {
                  writer.delete(entry);
                  F fileCopy = file.copyWithoutStats();

                  if (deletedFiles.contains(file)) {
                    LOG.warn(
                        "Deleting a duplicate path from manifest {}: {}",
                        manifest.path(),
                        file.location());
                    duplicateDeleteCount.incrementAndGet();
                  } else {
                    // only add the file to deletes if it is a new delete
                    // this keeps the snapshot summary accurate for non-duplicate data
                    deletedFiles.add(fileCopy);
                  }
                } else {
                  writer.existing(entry);
                }

              } else {
                writer.existing(entry);
              }
            });
      } catch (IOException e) {

View on GitHub (pinned to 86d9c8fc54)