{"record":{"id":"0ed7556f368d4aa7","repo":"apache/iceberg","slug":"cannot-commit-missing-data-files-s","errorCode":null,"errorMessage":"Cannot commit, missing data files: %s","messagePattern":"Cannot commit, missing data files: (.+?)","errorType":"validation","errorClass":"ValidationException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java","lineNumber":857,"sourceCode":"\n    ManifestGroup matchingDeletesGroup =\n        new ManifestGroup(ops().io(), manifests, ImmutableList.of())\n            .filterManifestEntries(\n                entry ->\n                    entry.status() != ManifestEntry.Status.ADDED\n                        && newSnapshots.contains(entry.snapshotId())\n                        && requiredDataFiles.contains(entry.file().location()))\n            .specsById(base.specsById())\n            .ignoreExisting();\n\n    if (conflictDetectionFilter != null) {\n      matchingDeletesGroup.filterData(conflictDetectionFilter);\n    }\n\n    try (CloseableIterator<ManifestEntry<DataFile>> deletes =\n        matchingDeletesGroup.entries().iterator()) {\n      if (deletes.hasNext()) {\n        throw new ValidationException(\n            \"Cannot commit, missing data files: %s\",\n            Iterators.toString(\n                Iterators.transform(deletes, entry -> entry.file().location().toString())));\n      }\n\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to validate required files exist\", e);\n    }\n  }\n\n  // validates there are no concurrently added DVs for referenced data files\n  protected void validateAddedDVs(\n      TableMetadata base,\n      Long startingSnapshotId,\n      Expression conflictDetectionFilter,\n      Snapshot parent) {\n    // skip if there is no current table state or this operation doesn't add new DVs\n    if (parent == null || dvsByReferencedFile.isEmpty()) {","sourceCodeStart":839,"sourceCodeEnd":875,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L839-L875","documentation":"ValidationException thrown during commit when data files referenced by delete files (equality or position deletes) no longer exist — i.e., some referenced data files are missing from the current table state. Iceberg aborts because applying the deletes would be against data files that have been removed or rewritten by a concurrent operation. It usually indicates conflicting concurrent writers or an operation based on a stale snapshot.","triggerScenarios":"Committing RowDelta/operation with conflictDetectionFilter set while concurrent commits deleted the data files this operation's delete files reference; detected via matchingDeletesGroup containing those missing files.","commonSituations":"Spark MERGE/DELETE running concurrently with compaction (rewriteDataFiles) that removed the referenced files; writer holding a stale snapshot for a long time; expired snapshots removing referenced files.","solutions":["Refresh the table and restart the operation against the current snapshot so delete files reference existing data files.","Re-run the MERGE/DELETE after the compaction job finishes; sequence rewrite and DML jobs.","Use validation (conflictDetectionFilter) correctly and retry the whole job rather than forcing the commit.","Avoid snapshot expiration of files still referenced by in-flight operations."],"exampleFix":"// before\nRowDelta rd = table.newRowDelta().addDeletes(deleteFile);\nrd.conflictDetectionFilter(filter);\nrd.validateNoConflictingDeletes(startSnapshotId);\nrd.commit(); // Cannot commit, missing data files\n// after\ntable.refresh();\nRowDelta rd = table.newRowDelta().addDeletes(recomputedDeleteFile); // recompute deletes on current snapshot\ntable.currentSnapshot(); // ensure base data files exist in current state\nrd.commit();","handlingStrategy":"retry","validationCode":"table.refresh();\n// verify referenced data files still exist before committing deletes\nSnapshot current = table.currentSnapshot();","typeGuard":null,"tryCatchPattern":"try {\n  rowDelta.commit();\n} catch (ValidationException e) {\n  table.refresh();\n  // recompute delete files against current data files and retry\n}","preventionTips":["Recompute delete files on the current snapshot before commit","Don't run compaction/rewrite concurrently with MERGE/DELETE that reference the same files","Avoid expiring snapshots that in-flight operations depend on"],"tags":["concurrency","stale-snapshot","commit-validation","delete-files"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}