apache/iceberg · error · CommitFailedException

Failed to acquire lock on file: %s with owner: %s

Error message

Failed to acquire lock on file: %s with owner: %s

What it means

Hadoop commits rename a new metadata file to its final versioned name; a lock manager (default: FileSystemLockManager) serializes this. renameToFinal throws CommitFailedException when acquire() returns false, meaning another writer holds the lock on the destination metadata file, or when the version already exists. This is the expected optimistic-concurrency failure signal for Hadoop tables.

Source

Thrown at core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java:364

      } catch (IOException io) {
        LOG.warn("Error trying to recover the latest version number for {}", versionHintFile, io);
        return 0;
      }
    }
  }

  /**
   * Renames the source file to destination, using the provided file system. If the rename failed,
   * an attempt will be made to delete the source file.
   *
   * @param fs the filesystem used for the rename
   * @param src the source file
   * @param dst the destination file
   */
  private void renameToFinal(FileSystem fs, Path src, Path dst, int nextVersion) {
    try {
      if (!lockManager.acquire(dst.toString(), src.toString())) {
        throw new CommitFailedException(
            "Failed to acquire lock on file: %s with owner: %s", dst, src);
      }

      if (fs.exists(dst)) {
        CommitFailedException cfe =
            new CommitFailedException("Version %d already exists: %s", nextVersion, dst);
        RuntimeException re = tryDelete(src);
        if (re != null) {
          cfe.addSuppressed(re);
        }

        throw cfe;
      }

      if (!fs.rename(src, dst)) {
        CommitFailedException cfe =
            new CommitFailedException("Failed to commit changes using rename: %s", dst);
        RuntimeException re = tryDelete(src);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Retry the commit after a backoff — CommitFailedException means another writer won this round
2. Remove stale lock files only after confirming no writer is active (check lock owner info)
3. Configure a shared, consistent LockManager implementation for all writers
4. Check whether the destination version file already exists; refresh and rebase your commit on the latest metadata

Example fix

// before
ops.commit(base, metadata); // throws on lock contention
// after
try {
  ops.commit(base, metadata);
} catch (CommitFailedException e) {
  ops.refresh();
  TableMetadata rebased = applyChanges(ops.current());
  ops.commit(ops.current(), rebased);
}

// pre-check lock availability where the LockManager exposes it
if (!lockManager.tryLock(dst.toString())) { /* back off and retry later */ }

try { ops.commit(base, metadata); } catch (CommitFailedException e) { backoff(); ops.refresh(); /* rebase and retry */ }

When it happens

Trigger: Two writers committing simultaneously to the same Hadoop table; a stale lock left behind by a crashed writer; lock-manager misconfiguration so different writers use inconsistent lock storage; the target version file already existing from a prior commit.

Common situations: Concurrent Spark/Flink jobs appending to a table stored on HDFS/local FS; leftover .lock files after a killed job; custom LockManager with a shared store unreachable causing failed acquisition.

Related errors

• Failed to release lock on file: {} with owner: {}
• Cannot commit %s due to unexpected exception
• Fail to acquire lock %s to commit new metadata at %s
• Cannot commit %s because base metadata location '%s' is not
• Cannot commit %s because Glue detected concurrent update

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/62974e8482afdb15. Report an issue: GitHub.