{"record":{"id":"c6c91d4bc82927e2","repo":"apache/iceberg","slug":"found-conflicting-files-that-can-contain-records-m","errorCode":null,"errorMessage":"Found conflicting files that can contain records matching partitions %s: %s","messagePattern":"Found conflicting files that can contain records matching partitions (.+?): (.+?)","errorType":"validation","errorClass":"ValidationException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java","lineNumber":370,"sourceCode":"  }\n\n  /**\n   * Validates that no files matching given partitions have been added to the table since a starting\n   * snapshot.\n   *\n   * @param base table metadata to validate\n   * @param startingSnapshotId id of the snapshot current at the start of the operation\n   * @param partitionSet a set of partitions to filter new conflicting data files\n   * @param parent ending snapshot on the lineage being validated\n   */\n  protected void validateAddedDataFiles(\n      TableMetadata base, Long startingSnapshotId, PartitionSet partitionSet, Snapshot parent) {\n    CloseableIterable<ManifestEntry<DataFile>> conflictEntries =\n        addedDataFiles(base, startingSnapshotId, null, partitionSet, parent);\n\n    try (CloseableIterator<ManifestEntry<DataFile>> conflicts = conflictEntries.iterator()) {\n      if (conflicts.hasNext()) {\n        throw new ValidationException(\n            \"Found conflicting files that can contain records matching partitions %s: %s\",\n            partitionSet,\n            Iterators.toString(\n                Iterators.transform(conflicts, entry -> entry.file().location().toString())));\n      }\n\n    } catch (IOException e) {\n      throw new UncheckedIOException(\n          String.format(\"Failed to validate no appends matching %s\", partitionSet), e);\n    }\n  }\n\n  /**\n   * Validates that no files matching a filter have been added to the table since a starting\n   * snapshot.\n   *\n   * @param base table metadata to validate\n   * @param startingSnapshotId id of the snapshot current at the start of the operation","sourceCodeStart":352,"sourceCodeEnd":388,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L352-L388","documentation":"A ValidationException raised by validateAddedDataFiles (partition-based conflict check). Before committing, the producer scans manifests added since the starting snapshot for data files in the conflicting partitions; if any exist, this concurrent-append conflict is reported with the partition set and the offending file locations.","triggerScenarios":"Two writers append data files to the same partitions concurrently (or sequentially after the snapshot this producer was based on) while both validate no conflicting appends — e.g. MergeOnEncrypted/compaction or delete commit validating no new files added to affected partitions since its starting snapshot.","commonSituations":"Concurrent Spark/Flink jobs writing to the same partition; DELETE/UPDATE statements racing with batch appends; retrying a stale commit after another job already appended.","solutions":["Rerun the failed operation against the latest snapshot so validation passes","Serialize writers per partition (e.g. use a lock, or write to distinct partitions)","Retry with Refresh: reload the table and rebuild the producer/plan from current metadata","Enable a different isolation level if the workload allows weaker validation (e.g. disable delete validation properties when safe)"],"exampleFix":"// before: stale producer fails validation\ntable.newRowDelta()... // built long ago\n.commit();\n// after: refresh table before rebuilding and committing\nTable refreshed = catalog.loadTable(tableIdentifier);\nrefreshed.newRowDelta()... // rebuild from refreshed snapshot\n.commit();","handlingStrategy":"retry","validationCode":"// detect conflicting appends before commit\nSnapshot current = table.currentSnapshot();\nif (!current.snapshotId().equals(plannedBaseSnapshotId)) { /* refresh and re-plan: another writer appended */ }","typeGuard":null,"tryCatchPattern":"try { producer.commit(); } catch (ValidationException e) { if (e.getMessage().startsWith(\"Found conflicting files\")) { table.refresh(); /* rebuild producer */ } else throw e; }","preventionTips":["Rebuild producers from a freshly loaded table immediately before commit","Avoid long gaps between scan planning and commit","Partition workloads so writers touch disjoint partitions","Use table-level coordination (locks or catalog-level serialization) for hot partitions"],"tags":["concurrency","conflict-detection","validation","snapshot"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}