What causes "Fail to acquire lock %s to commit new metadata at %s" in apache/iceberg?

Find and stop the competing writer holding the lock, or wait for it to finish and retry the commit.

apache/iceberg · error · IllegalStateException

Fail to acquire lock %s to commit new metadata at %s

Error message

Fail to acquire lock %s to commit new metadata at %s

What it means

GlueTableOperations.lock throws IllegalStateException when the configured LockManager (custom commit lock implementation) fails to acquire the commit lock for the table before writing new metadata. Iceberg requires the lock to guard against concurrent commits; failing to acquire it aborts the commit before any metadata is written.

Source

Thrown at aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java:258

                      .build())
              .build());
      return true;
    }

    return false;
  }

  private void cleanupGlueTempTableIfNecessary(
      boolean glueTempTableCreated, CommitStatus commitStatus) {
    if (glueTempTableCreated && commitStatus != CommitStatus.SUCCESS) {
      glue.deleteTable(
          DeleteTableRequest.builder().databaseName(databaseName).name(tableName).build());
    }
  }

  private void lock(String newMetadataLocation) {
    if (lockManager != null && !lockManager.acquire(commitLockEntityId, newMetadataLocation)) {
      throw new IllegalStateException(
          String.format(
              "Fail to acquire lock %s to commit new metadata at %s",
              commitLockEntityId, newMetadataLocation));
    }
  }

  private void checkMetadataLocation(Table glueTable, TableMetadata base) {
    String glueMetadataLocation =
        glueTable != null ? glueTable.parameters().get(METADATA_LOCATION_PROP) : null;
    String baseMetadataLocation = base != null ? base.metadataFileLocation() : null;
    if (!Objects.equals(baseMetadataLocation, glueMetadataLocation)) {
      throw new CommitFailedException(
          "Cannot commit %s because base metadata location '%s' is not same as the current Glue location '%s'",
          tableName(), baseMetadataLocation, glueMetadataLocation);
    }
  }

  private Table getGlueTable() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

Find and stop the competing writer holding the lock, or wait for it to finish and retry the commit.
Clear the stale lock left by a crashed job — use the LockManager's break/force-release mechanism (or delete the lock record, e.g. the DynamoDB item) once you confirm the holder is dead.
Configure lock timeouts/TTL so locks from dead processes expire automatically.
Verify the LockManager backend is reachable and IAM permissions allow read/write on the lock entity.

Example fix

// before
// custom LockManager without TTL; crashed job left the lock forever
new InMemoryLockManager();

// after
// use a lock backend with TTL so stale locks expire
DynamoDbLockManager lockManager = new DynamoDbLockManager();
lockManager.initialize("iceberg-commit-locks"); // locks expire if holder dies

Defensive patterns

Strategy: retry

Validate before calling

// before committing, verify the lock entity is free
if (!lockManager.isAcquired(commitLockEntityId)) {
  LOG.info("Lock {} is held by another writer, deferring commit", commitLockEntityId);
}

Try / catch

try {
  table.refresh(); // triggers doCommit -> lock internally
} catch (IllegalStateException e) {
  if (e.getMessage().startsWith("Fail to acquire lock")) {
    LOG.error("Commit lock {} held elsewhere; check for stale locks or competing writers", commitLockEntityId);
  }
  throw e;
}

Prevention

Use a LockManager backend with TTLs (e.g. DynamoDB) so crashed jobs release locks automatically
Ensure only one writer pipeline commits to a table at a time
Implement a documented stale-lock break procedure for dead holders
Verify IAM permissions on the lock table before starting commits

When it happens

Trigger: doCommit calls lock(newMetadataLocation) and lockManager.acquire(commitLockEntityId, location) returns false — the lock is already held by another process, or the LockManager backend (e.g. DynamoDB) is unavailable/misconfigured.

Common situations: Two Spark jobs committing to the same table simultaneously; a crashed job that never released its lock (stale lock); LockManager table unavailable or IAM denied; lock TTL/heartbeat misconfiguration.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/9f9afce5944da7c3. Report an issue: GitHub.