What causes "Cannot commit %s due to unexpected exception" in apache/iceberg?

Retry the write: reload the table (catalog.loadTable) to get fresh state and re-apply the operation — Iceberg commits are designed for optimistic retry.

apache/iceberg · error · CommitFailedException

Cannot commit %s due to unexpected exception

Error message

Cannot commit %s due to unexpected exception

What it means

GlueTableOperations.doCommit throws CommitFailedException when the metadata persistence to Glue failed with a definitive (FAILURE) commit status. The underlying exception (persistFailure) is attached as the cause; callers should treat this like any Iceberg commit conflict/failure and retry the operation from a fresh table state.

Source

Thrown at aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java:187

      if (!isAwsServiceException || retryDetector.retried()) {
        LOG.warn(
            "Received unexpected failure when committing to {}, validating if commit ended up succeeding.",
            fullTableName,
            persistFailure);
        commitStatus = checkCommitStatus(newMetadataLocation, metadata);
      }

      // If we got an AWS exception we would usually handle, but find we
      // succeeded on a retry that threw an exception, skip the exception.
      if (commitStatus != CommitStatus.SUCCESS && isAwsServiceException) {
        handleAWSExceptions((AwsServiceException) persistFailure);
      }

      switch (commitStatus) {
        case SUCCESS:
          break;
        case FAILURE:
          throw new CommitFailedException(
              persistFailure, "Cannot commit %s due to unexpected exception", tableName());
        case UNKNOWN:
          throw new CommitStateUnknownException(persistFailure);
      }
    } finally {
      cleanupMetadataAndUnlock(commitStatus, newMetadataLocation);
      cleanupGlueTempTableIfNecessary(glueTempTableCreated, commitStatus);
    }
  }

  /**
   * Validate the Glue table is Iceberg table by checking its parameters. If the table properties
   * check does not pass, for Iceberg it is equivalent to not having a table in the catalog. We
   * throw a {@link NoSuchIcebergTableException} in that case.
   *
   * @param table glue table
   * @param fullName full table name for logging
   * @throws NoSuchIcebergTableException if the table is not an Iceberg table

View on GitHub (pinned to 86d9c8fc54)

Solutions

Retry the write: reload the table (catalog.loadTable) to get fresh state and re-apply the operation — Iceberg commits are designed for optimistic retry.
Inspect the persistFailure cause (printed with the exception) to distinguish a conflict (retry-able) from a validation/permission error (fix required).
Reduce concurrent writers on the table or serialize commits through a single job/lock.
Check Glue service quotas/throttling and add backoff between commits; verify Lake Formation permissions for updates.

Example fix

// before
catalog.loadTable(ident).append(df); // throws CommitFailedException on conflict

// after
int attempts = 3;
while (true) {
  try {
    catalog.loadTable(ident).append(df); // reload inside loop = fresh snapshot
    break;
  } catch (CommitFailedException e) {
    if (--attempts == 0) throw e;
  }
}

Defensive patterns

Strategy: retry

Validate before calling

// no pre-call validation possible; conflicts happen at commit time
// ensure table exists and writer count is bounded before writing
if (!catalog.tableExists(ident)) {
  throw new IllegalStateException("Table missing before commit: " + ident);
}

Try / catch

int maxAttempts = 3;
for (int i = 0; i < maxAttempts; i++) {
  try {
    catalog.loadTable(ident).append(df); // fresh load each attempt
    break;
  } catch (CommitFailedException e) {
    if (i == maxAttempts - 1) throw e;
    Uninterruptibles.sleepUninterruptibly(100L * (i + 1), TimeUnit.MILLISECONDS);
  }
}

Prevention

Always retry with a freshly loaded table — CommitFailedException means state changed under you
Limit concurrent writers per table or serialize commits
Inspect the attached persistFailure cause to separate conflicts from permission/validation errors
Watch Glue throttling quotas under heavy write load and add backoff

When it happens

Trigger: A Glue UpdateTable call during commit fails definitively — e.g. OptimisticLockException style conflicts when another writer committed concurrently, validation errors on the table update, throttling, or AWS service errors that are deterministically failed rather than unknown-outcome.

Common situations: Multiple concurrent writers committing to the same table (classic optimistic concurrency conflict); Glue API throttling under heavy write load; Lake Formation rejecting the update due to permissions; transient AWS outages.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/213e038eef4330c2. Report an issue: GitHub.