Commit state unknown, cannot clean up files that may have be

Error message

Commit state unknown, cannot clean up files that may have been committed

What it means

When committing a rewrite in BaseRewriteDataFilesAction.replaceDataFiles(), a CommitStateUnknownException means the commit may or may not have succeeded. Because the added data files might already be committed to the table, they cannot be safely deleted, so the action logs this warning and rethrows the exception. Deleting them could corrupt the table by removing committed files.

Source

Thrown at core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java:310

      iterator.forEachRemaining(
          task -> {
            StructLikeWrapper structLike = partitionWrapper.copyFor(task.file().partition());
            tasksGroupedByPartition.put(structLike, task);
          });
    } catch (IOException e) {
      LOG.warn("Failed to close task iterator", e);
    }
    return tasksGroupedByPartition.asMap();
  }

  private void replaceDataFiles(
      Iterable<DataFile> deletedDataFiles,
      Iterable<DataFile> addedDataFiles,
      long startingSnapshotId) {
    try {
      doReplace(deletedDataFiles, addedDataFiles, startingSnapshotId);
    } catch (CommitStateUnknownException e) {
      LOG.warn("Commit state unknown, cannot clean up files that may have been committed", e);
      throw e;
    } catch (Exception e) {
      if (e instanceof CleanableFailure) {
        LOG.warn("Failed to commit rewrite, cleaning up rewritten files", e);
        Tasks.foreach(Iterables.transform(addedDataFiles, ContentFile::location))
            .noRetry()
            .suppressFailureWhenFinished()
            .onFailure((location, exc) -> LOG.warn("Failed to delete: {}", location, exc))
            .run(fileIO::deleteFile);
      }

      throw e;
    }
  }

  @VisibleForTesting
  void doReplace(
      Iterable<DataFile> deletedDataFiles,

View on GitHub (pinned to 86d9c8fc54)

Solutions

Determine actual commit state: refresh the table and check the latest snapshot/manifests for the rewritten files before taking any cleanup action.
Do NOT delete the new data files when state is unknown — rely on removeOrphanFiles with a safe age threshold later.
Fix the underlying catalog reliability issue (timeouts, lost responses) that caused the indeterminate commit.
Re-run the rewrite if the commit did not land; Iceberg's idempotent design makes a re-run safe.

Defensive patterns

Strategy: try-catch

Validate before calling

try {
  table.refresh();
} catch (RuntimeException e) {
  throw new IllegalStateException("Cannot verify commit state; treat new files as possibly committed and do not delete", e);
}

Try / catch

try {
  replace(deleted, added, snapshotId);
} catch (CommitStateUnknownException e) {
  table.refresh();
  // keep added files; verify via snapshot inspection, clean later with removeOrphanFiles
  LOG.warn("Commit state unknown; files preserved for safety", e);
}

Prevention

Never delete rewritten output files on CommitStateUnknownException
Use catalogs that report commit outcome deterministically
Schedule RemoveOrphanFiles with a conservative olderThan threshold
Reduce catalog timeouts/failures that cause indeterminate commits

When it happens

Trigger: doReplace() → table transaction commit throws CommitStateUnknownException (commit outcome indeterminate) during replaceDataFiles() in a RewriteDataFiles action.

Common situations: Catalog/network failures at exactly the commit boundary (response lost after the commit request was sent); custom catalog implementations that cannot determine commit outcome; object-store timeouts during commit finalization.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/bd86042e97f87e49. Report an issue: GitHub.