{"record":{"id":"8fd5619a0cec3abf","repo":"apache/iceberg","slug":"cannot-commit-changes-based-on-stale-table-metadat","errorCode":null,"errorMessage":"Cannot commit changes based on stale table metadata","messagePattern":"Cannot commit changes based on stale table metadata","errorType":"exception","errorClass":"CommitFailedException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java","lineNumber":134,"sourceCode":"        ver += 1;\n        metadataFile = nextMetadataFile;\n        nextMetadataFile = getMetadataFile(ver + 1);\n      }\n\n      updateVersionAndMetadata(ver, metadataFile.toString());\n\n      this.shouldRefresh = false;\n      return currentMetadata;\n    } catch (IOException e) {\n      throw new RuntimeIOException(e, \"Failed to refresh the table\");\n    }\n  }\n\n  @Override\n  public void commit(TableMetadata base, TableMetadata metadata) {\n    Pair<Integer, TableMetadata> current = versionAndMetadata();\n    if (base != current.second()) {\n      throw new CommitFailedException(\"Cannot commit changes based on stale table metadata\");\n    }\n\n    if (base == metadata) {\n      LOG.info(\"Nothing to commit.\");\n      return;\n    }\n\n    Preconditions.checkArgument(\n        base == null || base.location().equals(metadata.location()),\n        \"Hadoop path-based tables cannot be relocated\");\n    Preconditions.checkArgument(\n        !metadata.properties().containsKey(TableProperties.WRITE_METADATA_LOCATION),\n        \"Hadoop path-based tables cannot relocate metadata\");\n\n    String codecName =\n        metadata.property(\n            TableProperties.METADATA_COMPRESSION, TableProperties.METADATA_COMPRESSION_DEFAULT);\n    TableMetadataParser.Codec codec = TableMetadataParser.Codec.fromName(codecName);","sourceCodeStart":116,"sourceCodeEnd":152,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java#L116-L152","documentation":"commit() requires that the base TableMetadata passed in is exactly the same object (same version) the operations instance currently holds. If the caller's base is stale relative to the on-disk table, the commit would silently overwrite concurrent changes, so CommitStateUnknown/CommitFailedException is thrown. Callers must refresh and retry.","triggerScenarios":"Calling ops.commit(base, metadata) after another process committed a new snapshot so base != current(); using a Table reference whose metadata was loaded before an external commit; calling commit without refresh() after a prior commit from the same reference.","commonSituations":"Two Spark jobs writing to the same Hadoop-backed table concurrently; reusing a cached Table object across a long job while other writers commit; skipping refresh() between consecutive commits from one process.","solutions":["Call table.refresh() to get latest metadata, reapply your changes to the fresh base, then retry commit","Use a catalog with atomic commit support instead of raw HadoopTables for multi-writer workloads","Serialize writers externally (lock) when using HadoopTables with concurrent writers","Wrap commit in a retry loop catching CommitFailedException with bounded attempts"],"exampleFix":"// before\nops.commit(staleBase, updatedMetadata);\n// after\nboolean committed = false;\nfor (int i = 0; i < 3 && !committed; i++) {\n  try {\n    ops.commit(ops.current(), updatedFor(ops.current()));\n    committed = true;\n  } catch (CommitFailedException e) {\n    ops.refresh();\n  }\n}","handlingStrategy":"retry","validationCode":"if (ops.current() != base) { ops.refresh(); /* rebase changes */ }","typeGuard":null,"tryCatchPattern":"try { ops.commit(base, metadata); } catch (CommitFailedException e) { ops.refresh(); /* rebase and retry */ }","preventionTips":["Always refresh() before commit when the table may have changed externally","Wrap commits in bounded retry loops against CommitFailedException","Avoid long-lived Table references in multi-writer environments","Prefer catalogs with atomic commits over HadoopTables for concurrent writers"],"tags":["concurrency","commit","optimistic-locking"],"backgroundTag":"commit-conflict-stale-metadata","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}