{"record":{"id":"63ab4db930ee2cdd","repo":"apache/iceberg","slug":"replaced-and-created-manifests-must-have-the-same","errorCode":null,"errorMessage":"Replaced and created manifests must have the same number of active files: %d (new), %d (old)","messagePattern":"Replaced and created manifests must have the same number of active files: (.+?) \\(new\\), (.+?) \\(old\\)","errorType":"validation","errorClass":"ValidationException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java","lineNumber":310,"sourceCode":"        .ifPresent(\n            manifest -> {\n              throw new ValidationException(\n                  \"Deleted manifest %s could not be found in the latest snapshot %d\",\n                  manifest.path(), currentSnapshotID);\n            });\n  }\n\n  private void validateFilesCounts() {\n    Iterable<ManifestFile> createdManifests =\n        Iterables.concat(newManifests, addedManifests, rewrittenAddedManifests);\n    int createdManifestsFilesCount = activeFilesCount(createdManifests);\n\n    Iterable<ManifestFile> replacedManifests =\n        Iterables.concat(rewrittenManifests, deletedManifests);\n    int replacedManifestsFilesCount = activeFilesCount(replacedManifests);\n\n    if (createdManifestsFilesCount != replacedManifestsFilesCount) {\n      throw new ValidationException(\n          \"Replaced and created manifests must have the same number of active files: %d (new), %d (old)\",\n          createdManifestsFilesCount, replacedManifestsFilesCount);\n    }\n  }\n\n  private int activeFilesCount(Iterable<ManifestFile> manifests) {\n    int activeFilesCount = 0;\n\n    for (ManifestFile manifest : manifests) {\n      Preconditions.checkNotNull(\n          manifest.addedFilesCount(), \"Missing file counts in %s\", manifest.path());\n      Preconditions.checkNotNull(\n          manifest.existingFilesCount(), \"Missing file counts in %s\", manifest.path());\n      activeFilesCount += manifest.addedFilesCount();\n      activeFilesCount += manifest.existingFilesCount();\n    }\n\n    return activeFilesCount;","sourceCodeStart":292,"sourceCodeEnd":328,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java#L292-L328","documentation":"BaseRewriteManifests validates, after rewriting, that the manifests it created hold exactly the same number of active (live) data files as the manifests it replaced or deleted. If the writer dropped, skipped, or duplicated files during rewrite, the table's file accounting would silently change, so a ValidationException is thrown before commit.","triggerScenarios":"A custom ManifestsWriter or custom rewrite logic (e.g. subclassing BaseRewriteManifests via an engine integration) filters out data files while copying entries, or fails to add all entries to the new manifest writers, so createdManifestsFilesCount != replacedManifestsFilesCount.","commonSituations":"Custom rewrite-manifests procedures in Spark/Flink integrations that add filtering or partial rewrites; bugs in third-party catalog or writer plugins; running a rewrite across snapshots that changed concurrently so entries read as deleted.","solutions":["Fix the rewrite logic so every active data file entry from each replaced manifest is written to exactly one new manifest (deleted entries may be skipped)","Ensure the writer is not closed/flushed before all entries are added — check addEntry/close ordering in the custom Writer","Retry the rewrite against a fresh table snapshot, since the source manifests may have changed mid-rewrite","If seen in third-party tooling, report/upgrade the plugin rather than bypassing the validation"],"exampleFix":"// before: filter while rewriting\nfor (ManifestEntry<DataFile> e : manifest.entries()) {\n  if (e.file().fileSizeInBytes() > threshold) writer.add(e); // drops small files\n}\n// after: rewrite preserves all active files; filter via spec later\nfor (ManifestEntry<DataFile> e : manifest.entries()) {\n  if (e.status() == ManifestEntry.Status.DELETED) continue;\n  writer.add(e);\n}","handlingStrategy":"validation","validationCode":"long created = newManifests.stream().mapToLong(ManifestFile::existingFilesCount).sum();\nlong replaced = replacedManifests.stream().mapToLong(ManifestFile::existingFilesCount).sum();\nif (created != replaced) throw new IllegalStateException(\"rewrite would drop files: \" + created + \" vs \" + replaced);","typeGuard":null,"tryCatchPattern":"try { table.rewriteManifests(); } catch (ValidationException e) { LOG.error(\"Manifest rewrite invariant broken\", e); }","preventionTips":["Never filter data files during manifest rewrite; filter via delete/expiry operations instead","Test custom writers against manifests with DELETED and EXISTING entries","Keep writer add/close ordering correct in custom Writer implementations"],"tags":["iceberg","manifest-rewrite","validation"],"backgroundTag":"internal-invariant-violation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}