apache/iceberg · error · ValidationException

Replaced and created manifests must have the same number of

Error message

Replaced and created manifests must have the same number of active files: %d (new), %d (old)

What it means

BaseRewriteManifests validates, after rewriting, that the manifests it created hold exactly the same number of active (live) data files as the manifests it replaced or deleted. If the writer dropped, skipped, or duplicated files during rewrite, the table's file accounting would silently change, so a ValidationException is thrown before commit.

Source

Thrown at core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java:310

        .ifPresent(
            manifest -> {
              throw new ValidationException(
                  "Deleted manifest %s could not be found in the latest snapshot %d",
                  manifest.path(), currentSnapshotID);
            });
  }

  private void validateFilesCounts() {
    Iterable<ManifestFile> createdManifests =
        Iterables.concat(newManifests, addedManifests, rewrittenAddedManifests);
    int createdManifestsFilesCount = activeFilesCount(createdManifests);

    Iterable<ManifestFile> replacedManifests =
        Iterables.concat(rewrittenManifests, deletedManifests);
    int replacedManifestsFilesCount = activeFilesCount(replacedManifests);

    if (createdManifestsFilesCount != replacedManifestsFilesCount) {
      throw new ValidationException(
          "Replaced and created manifests must have the same number of active files: %d (new), %d (old)",
          createdManifestsFilesCount, replacedManifestsFilesCount);
    }
  }

  private int activeFilesCount(Iterable<ManifestFile> manifests) {
    int activeFilesCount = 0;

    for (ManifestFile manifest : manifests) {
      Preconditions.checkNotNull(
          manifest.addedFilesCount(), "Missing file counts in %s", manifest.path());
      Preconditions.checkNotNull(
          manifest.existingFilesCount(), "Missing file counts in %s", manifest.path());
      activeFilesCount += manifest.addedFilesCount();
      activeFilesCount += manifest.existingFilesCount();
    }

    return activeFilesCount;

View on GitHub (pinned to 86d9c8fc54)

Solutions

Fix the rewrite logic so every active data file entry from each replaced manifest is written to exactly one new manifest (deleted entries may be skipped)
Ensure the writer is not closed/flushed before all entries are added — check addEntry/close ordering in the custom Writer
Retry the rewrite against a fresh table snapshot, since the source manifests may have changed mid-rewrite
If seen in third-party tooling, report/upgrade the plugin rather than bypassing the validation

Example fix

// before: filter while rewriting
for (ManifestEntry<DataFile> e : manifest.entries()) {
  if (e.file().fileSizeInBytes() > threshold) writer.add(e); // drops small files
}
// after: rewrite preserves all active files; filter via spec later
for (ManifestEntry<DataFile> e : manifest.entries()) {
  if (e.status() == ManifestEntry.Status.DELETED) continue;
  writer.add(e);
}

Defensive patterns

Strategy: validation

Validate before calling

long created = newManifests.stream().mapToLong(ManifestFile::existingFilesCount).sum();
long replaced = replacedManifests.stream().mapToLong(ManifestFile::existingFilesCount).sum();
if (created != replaced) throw new IllegalStateException("rewrite would drop files: " + created + " vs " + replaced);

Try / catch

try { table.rewriteManifests(); } catch (ValidationException e) { LOG.error("Manifest rewrite invariant broken", e); }

Prevention

Never filter data files during manifest rewrite; filter via delete/expiry operations instead
Test custom writers against manifests with DELETED and EXISTING entries
Keep writer add/close ordering correct in custom Writer implementations

When it happens

Trigger: A custom ManifestsWriter or custom rewrite logic (e.g. subclassing BaseRewriteManifests via an engine integration) filters out data files while copying entries, or fails to add all entries to the new manifest writers, so createdManifestsFilesCount != replacedManifestsFilesCount.

Common situations: Custom rewrite-manifests procedures in Spark/Flink integrations that add filtering or partial rewrites; bugs in third-party catalog or writer plugins; running a rewrite across snapshots that changed concurrently so entries read as deleted.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/63ab4db930ee2cdd. Report an issue: GitHub.