{"record":{"id":"bd86042e97f87e49","repo":"apache/iceberg","slug":"commit-state-unknown-cannot-clean-up-files-that-m","errorCode":null,"errorMessage":"Commit state unknown, cannot clean up files that may have been committed","messagePattern":"Commit state unknown, cannot clean up files that may have been committed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java","lineNumber":310,"sourceCode":"      iterator.forEachRemaining(\n          task -> {\n            StructLikeWrapper structLike = partitionWrapper.copyFor(task.file().partition());\n            tasksGroupedByPartition.put(structLike, task);\n          });\n    } catch (IOException e) {\n      LOG.warn(\"Failed to close task iterator\", e);\n    }\n    return tasksGroupedByPartition.asMap();\n  }\n\n  private void replaceDataFiles(\n      Iterable<DataFile> deletedDataFiles,\n      Iterable<DataFile> addedDataFiles,\n      long startingSnapshotId) {\n    try {\n      doReplace(deletedDataFiles, addedDataFiles, startingSnapshotId);\n    } catch (CommitStateUnknownException e) {\n      LOG.warn(\"Commit state unknown, cannot clean up files that may have been committed\", e);\n      throw e;\n    } catch (Exception e) {\n      if (e instanceof CleanableFailure) {\n        LOG.warn(\"Failed to commit rewrite, cleaning up rewritten files\", e);\n        Tasks.foreach(Iterables.transform(addedDataFiles, ContentFile::location))\n            .noRetry()\n            .suppressFailureWhenFinished()\n            .onFailure((location, exc) -> LOG.warn(\"Failed to delete: {}\", location, exc))\n            .run(fileIO::deleteFile);\n      }\n\n      throw e;\n    }\n  }\n\n  @VisibleForTesting\n  void doReplace(\n      Iterable<DataFile> deletedDataFiles,","sourceCodeStart":292,"sourceCodeEnd":328,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java#L292-L328","documentation":"When committing a rewrite in BaseRewriteDataFilesAction.replaceDataFiles(), a CommitStateUnknownException means the commit may or may not have succeeded. Because the added data files might already be committed to the table, they cannot be safely deleted, so the action logs this warning and rethrows the exception. Deleting them could corrupt the table by removing committed files.","triggerScenarios":"doReplace() → table transaction commit throws CommitStateUnknownException (commit outcome indeterminate) during replaceDataFiles() in a RewriteDataFiles action.","commonSituations":"Catalog/network failures at exactly the commit boundary (response lost after the commit request was sent); custom catalog implementations that cannot determine commit outcome; object-store timeouts during commit finalization.","solutions":["Determine actual commit state: refresh the table and check the latest snapshot/manifests for the rewritten files before taking any cleanup action.","Do NOT delete the new data files when state is unknown — rely on removeOrphanFiles with a safe age threshold later.","Fix the underlying catalog reliability issue (timeouts, lost responses) that caused the indeterminate commit.","Re-run the rewrite if the commit did not land; Iceberg's idempotent design makes a re-run safe."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"try {\n  table.refresh();\n} catch (RuntimeException e) {\n  throw new IllegalStateException(\"Cannot verify commit state; treat new files as possibly committed and do not delete\", e);\n}","typeGuard":null,"tryCatchPattern":"try {\n  replace(deleted, added, snapshotId);\n} catch (CommitStateUnknownException e) {\n  table.refresh();\n  // keep added files; verify via snapshot inspection, clean later with removeOrphanFiles\n  LOG.warn(\"Commit state unknown; files preserved for safety\", e);\n}","preventionTips":["Never delete rewritten output files on CommitStateUnknownException","Use catalogs that report commit outcome deterministically","Schedule RemoveOrphanFiles with a conservative olderThan threshold","Reduce catalog timeouts/failures that cause indeterminate commits"],"tags":["commit","commit-state-unknown","rewrite","data-safety"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}