Failed to commit rewrite, cleaning up rewritten files

Error message

Failed to commit rewrite, cleaning up rewritten files

What it means

If a rewrite commit in BaseRewriteDataFilesAction.replaceDataFiles() fails with an exception implementing CleanableFailure (i.e. the commit definitively failed and the new files were not committed), the action logs this warning and deletes the newly written data files to avoid orphans. This is the safe-cleanup branch, distinct from the CommitStateUnknown case where files are kept.

Source

Thrown at core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java:314

          });
    } catch (IOException e) {
      LOG.warn("Failed to close task iterator", e);
    }
    return tasksGroupedByPartition.asMap();
  }

  private void replaceDataFiles(
      Iterable<DataFile> deletedDataFiles,
      Iterable<DataFile> addedDataFiles,
      long startingSnapshotId) {
    try {
      doReplace(deletedDataFiles, addedDataFiles, startingSnapshotId);
    } catch (CommitStateUnknownException e) {
      LOG.warn("Commit state unknown, cannot clean up files that may have been committed", e);
      throw e;
    } catch (Exception e) {
      if (e instanceof CleanableFailure) {
        LOG.warn("Failed to commit rewrite, cleaning up rewritten files", e);
        Tasks.foreach(Iterables.transform(addedDataFiles, ContentFile::location))
            .noRetry()
            .suppressFailureWhenFinished()
            .onFailure((location, exc) -> LOG.warn("Failed to delete: {}", location, exc))
            .run(fileIO::deleteFile);
      }

      throw e;
    }
  }

  @VisibleForTesting
  void doReplace(
      Iterable<DataFile> deletedDataFiles,
      Iterable<DataFile> addedDataFiles,
      long startingSnapshotId) {
    RewriteFiles rewriteFiles = table.newRewrite().validateFromSnapshot(startingSnapshotId);

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the logged CleanableFailure cause to understand why the commit definitively failed.
Re-run the RewriteDataFiles action after resolving the conflict (e.g. fewer concurrent writers, retry).
If conflicts are frequent, reduce file-group overlap or commit concurrently with other jobs less aggressively.
Check that the cleanup deletion itself succeeded — per-file delete failures are logged as 'Failed to delete' warnings; clean leftovers with RemoveOrphanFiles.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  replace(deleted, added, snapshotId);
} catch (CleanableFailure e) {
  added.forEach(file -> fileIO.deleteFile(file.location())); // definite failure: safe to clean up
  throw e;
} catch (CommitStateUnknownException e) {
  // do NOT delete added files; state unknown
  throw e;
}

Prevention

Implement CleanableFailure for definitive commit failures so cleanup can proceed
Schedule RemoveOrphanFiles to catch any delete failures during cleanup
Limit concurrent writers to reduce commit conflicts
Re-run rewrites after resolving the reported failure cause

When it happens

Trigger: doReplace() commit throws an exception implementing CleanableFailure during replaceDataFiles(); the action responds by deleting all addedDataFiles via FileIO with no retry.

Common situations: Validation failures at commit (concurrent table changes conflicting with the rewrite); optimistic-concurrency conflicts resolved as definite failures; deterministic commit rejections by the catalog.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/ec9384e20ade0e45. Report an issue: GitHub.