apache/iceberg · error · CommitFailedException

Cannot commit %s: concurrent update detected

Error message

Cannot commit %s: concurrent update detected

What it means

CommitFailedException thrown in DynamoDbTableOperations.doCommit when the conditional update failed (conditionCheckFailed) and the commit could not be confirmed as successful. Another writer updated the table's metadata location between refresh and commit, so the optimistic concurrency check failed.

Source

Thrown at aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:140

      commitStatus = CommitStatus.SUCCESS;
    } catch (CommitFailedException e) {
      // any explicit commit failures are passed up and out to the retry handler
      throw e;
    } catch (RuntimeException persistFailure) {
      boolean conditionCheckFailed = persistFailure instanceof ConditionalCheckFailedException;

      // If we got an exception we weren't expecting, or we got a ConditionalCheckFailedException
      // but retries were performed, attempt to reconcile the actual commit status.
      if (!conditionCheckFailed || retryDetector.retried()) {
        LOG.warn(
            "Received unexpected failure when committing to {}, validating if commit ended up succeeding.",
            fullTableName,
            persistFailure);
        commitStatus = checkCommitStatus(newMetadataLocation, metadata);
      }

      if (commitStatus != CommitStatus.SUCCESS && conditionCheckFailed) {
        throw new CommitFailedException(
            persistFailure, "Cannot commit %s: concurrent update detected", tableName());
      }

      switch (commitStatus) {
        case SUCCESS:
          break;
        case FAILURE:
          throw new CommitFailedException(
              persistFailure, "Cannot commit %s due to unexpected exception", tableName());
        case UNKNOWN:
          throw new CommitStateUnknownException(persistFailure);
      }
    } finally {
      try {
        if (commitStatus == CommitStatus.FAILURE) {
          // if anything went wrong, clean up the uncommitted metadata file
          io().deleteFile(newMetadataLocation);
        }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Refresh the table and retry the operation from the new base metadata
2. Serialize writers with DynamoDbLockManager or an external lock so only one commits at a time
3. Ensure the committing job's metadata snapshot is fresh (call refresh() right before commit)

Example fix

// before
Table t = catalog.loadTable(id);
t.append(df); // commit may fail under contention
// after
Table t = catalog.loadTable(id);
try {
  t.refresh();
  t.append(df);
} catch (CommitFailedException e) {
  t.refresh();
  t.append(df); // retry from fresh base
}

table.refresh(); // rebase on latest metadata right before writing

try {
  table.append(df);
} catch (CommitFailedException e) {
  table.refresh();
  table.append(df); // retry once from fresh metadata
}

When it happens

Trigger: Two writers commit to the same table concurrently; the PutItem condition on COL_METADATA_LOCATION fails, then checkCommitStatus does not return SUCCESS.

Common situations: Multiple Spark jobs or services writing to the same table simultaneously; long-running plan followed by commit racing a shorter job; retries after lost leases.

Related errors

• Cannot commit %s because base metadata location '%s' is not
• Cannot find default warehouse location: namespace %s does no
• Cannot create namespace %s: already exists
• Cannot find namespace %s
• Cannot delete non-empty namespace %s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f60524317c46607f. Report an issue: GitHub.