apache/iceberg · error · LockException

Failed to heartbeat for hive lock. %s

Error message

Failed to heartbeat for hive lock. %s

What it means

The lock's heartbeat thread hit an exception while heartbeating the Hive lock to the metastore, so the lock can no longer be considered valid. ensureActive surfaces this as a LockException so the commit is aborted rather than proceeding with a possibly-lost lock.

Source

Thrown at hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java:160

    // from the same JVM process, which would result in unnecessary HMS lock acquisition requests
    acquireJvmLock();

    // Getting HMS lock
    hmsLockId = Optional.of(acquireLock());

    // Starting heartbeat for the HMS lock
    heartbeat = new Heartbeat(metaClients, hmsLockId.get(), lockHeartbeatIntervalTime);
    heartbeat.schedule(exitingScheduledExecutorService);
  }

  @Override
  public void ensureActive() throws LockException {
    if (heartbeat == null) {
      throw new LockException("Lock is not active");
    }

    if (heartbeat.encounteredException != null) {
      throw new LockException(
          heartbeat.encounteredException,
          "Failed to heartbeat for hive lock. %s",
          heartbeat.encounteredException.getMessage());
    }
    if (!heartbeat.active()) {
      throw new LockException("Hive lock heartbeat thread not active");
    }
  }

  @Override
  public void unlock() {
    if (heartbeat != null) {
      heartbeat.cancel();
      exitingScheduledExecutorService.shutdown();
    }

    try {
      unlock(hmsLockId);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Retry the commit; the lock must be re-acquired after the heartbeat failure.
2. Increase hive.txn.timeout or ensure heartbeats fire well within it for long-running commits.
3. Check metastore availability/network stability during the commit window.

Example fix

// before
conf: lock-heartbeat-interval = 1h (longer than metastore lock timeout)

// after
conf: heartbeat interval set well below hive.txn.timeout (e.g. minutes),
retry commit on CommitFailedException

try {
  commit();
} catch (CommitFailedException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to heartbeat")) {
    // re-acquire lock, refresh view, retry with backoff
  }
}

When it happens

Trigger: Heartbeat thread's metastore heartbeatLock call threw (TException, connection loss, lock already expired/released); ensureActive reads heartbeat.encounteredException and rethrows it.

Common situations: Long commits exceeding lock timeout so the metastore reaped the lock; metastore outage during commit; heartbeat interval longer than metastore's lock timeout (hive.txn.timeout).

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Failed to acquire locks from metastore because the underlyin
• Hive lock heartbeat thread not active
• Metastore operation failed for %s.%s
• Could not acquire the lock on %s.%s, lock request ended in s
• Failed to heartbeat for lock: %d

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/98529de08d3bf112. Report an issue: GitHub.