{"record":{"id":"6a3fef9e23137126","repo":"apache/iceberg","slug":"failed-to-validate-required-files-exist","errorCode":null,"errorMessage":"Failed to validate required files exist","messagePattern":"Failed to validate required files exist","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java","lineNumber":864,"sourceCode":"                        && requiredDataFiles.contains(entry.file().location()))\n            .specsById(base.specsById())\n            .ignoreExisting();\n\n    if (conflictDetectionFilter != null) {\n      matchingDeletesGroup.filterData(conflictDetectionFilter);\n    }\n\n    try (CloseableIterator<ManifestEntry<DataFile>> deletes =\n        matchingDeletesGroup.entries().iterator()) {\n      if (deletes.hasNext()) {\n        throw new ValidationException(\n            \"Cannot commit, missing data files: %s\",\n            Iterators.toString(\n                Iterators.transform(deletes, entry -> entry.file().location().toString())));\n      }\n\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Failed to validate required files exist\", e);\n    }\n  }\n\n  // validates there are no concurrently added DVs for referenced data files\n  protected void validateAddedDVs(\n      TableMetadata base,\n      Long startingSnapshotId,\n      Expression conflictDetectionFilter,\n      Snapshot parent) {\n    // skip if there is no current table state or this operation doesn't add new DVs\n    if (parent == null || dvsByReferencedFile.isEmpty()) {\n      return;\n    }\n\n    Pair<List<ManifestFile>, Set<Long>> history =\n        validationHistory(\n            base,\n            startingSnapshotId,","sourceCodeStart":846,"sourceCodeEnd":882,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L846-L882","documentation":"UncheckedIOException wrapping an IOException that occurred while checking that data files required by delete files still exist at commit time. The existence validation could not finish because manifest scanning failed on I/O, so the commit aborts. The underlying cause is in the chained exception.","triggerScenarios":"Committing an operation that calls validateRequiredFilesExist (e.g., RowDelta with delete files) where iterating the manifest entries throws IOException from the underlying FileIO.","commonSituations":"Storage outages (S3/HDFS) during commit; expired credentials; data/manifest files deleted by concurrent snapshot expiration during validation.","solutions":["Retry the commit with a retry policy once the storage backend is reachable.","Fix the root IOException cause (network, credentials, missing files) from the exception chain.","Ensure snapshot expiration jobs are not deleting files concurrently with commits.","Refresh the table right before commit to keep the validation window short."],"exampleFix":"// before\nrowDelta.commit(); // UncheckedIOException: Failed to validate required files exist\n// after\nTasks.foreach(rowDelta::commit)\n    .retry(3).exponentialBackoff(100, 10000, 60000)\n    .throwFailureWhenFinished()\n    .run();","handlingStrategy":"retry","validationCode":"// pre-check required data file existence via FileIO\nfor (String path : referencedPaths) { io.newInputFile(path).exists(); }","typeGuard":null,"tryCatchPattern":"try {\n  rowDelta.commit();\n} catch (UncheckedIOException e) {\n  // inspect e.getCause(); retry once storage is healthy\n}","preventionTips":["Retry commits with backoff around storage faults","Keep expiration jobs isolated from conflicting commits","Refresh close to commit to shrink the validation window"],"tags":["io","commit-validation","unchecked-io-exception"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}