apache/iceberg · error · CommitFailedException

Failed to heartbeat for lock: %d

Error message

Failed to heartbeat for lock: %d

What it means

MetastoreLock runs a scheduled heartbeat task that periodically calls the metastore heartbeat RPC to keep the acquired lock alive. If the heartbeat fails with a TException or InterruptedException, the exception is recorded and a CommitFailedException with this message is thrown, signaling the lock may have been lost and the commit must be retried. The lock will typically expire (hive.txn.timeout) without heartbeats.

Source

Thrown at hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java:492

    Heartbeat(ClientPool<IMetaStoreClient, TException> hmsClients, long lockId, long intervalMs) {
      this.hmsClients = hmsClients;
      this.lockId = lockId;
      this.intervalMs = intervalMs;
      this.future = null;
    }

    @Override
    public void run() {
      try {
        hmsClients.run(
            client -> {
              client.heartbeat(0, lockId);
              return null;
            });
      } catch (TException | InterruptedException e) {
        this.encounteredException = e;
        throw new CommitFailedException(e, "Failed to heartbeat for lock: %d", lockId);
      }
    }

    public void schedule(ScheduledExecutorService scheduler) {
      future =
          scheduler.scheduleAtFixedRate(this, intervalMs / 2, intervalMs, TimeUnit.MILLISECONDS);
    }

    boolean active() {
      return future != null && !future.isCancelled();
    }

    public void cancel() {
      if (future != null) {
        future.cancel(false);
      }
    }
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Retry the commit; the CommitFailedException is the designed signal that the lock was lost and the commit is safe to re-attempt.
2. Set lock-heartbeat-interval-ms well below hive.txn.timeout so a missed heartbeat still leaves time to recover.
3. Fix metastore connectivity/uptime issues that interrupt heartbeats (network, metastore restarts, connection pools).
4. Check the recorded encounteredException cause for the underlying thrift error before tuning.

Example fix

// before
table.properties().set("lock-heartbeat-interval-ms", "600000"); // > txn timeout
// after
table.properties().set("lock-heartbeat-interval-ms", "30000"); // well below hive.txn.timeout

long heartbeatMs = Long.parseLong(table.properties().getOrDefault(
    "lock-heartbeat-interval-ms", "30000"));
long txnTimeoutMs = Long.parseLong(hiveConf.get("hive.txn.timeout"));
if (heartbeatMs * 2 >= txnTimeoutMs) {
  throw new IllegalArgumentException("heartbeat interval must be well below hive.txn.timeout");
}

try {
  table.commit(apply);
} catch (CommitFailedException e) {
  if (e.getMessage().startsWith("Failed to heartbeat for lock")) {
    // lock lost due to heartbeat failure; refresh and re-attempt
    backoffRetry(() -> { table.refresh(); table.commit(apply); });
  } else throw e;
}

When it happens

Trigger: Scheduled heartbeat of an acquired table lock fails: metastore becomes unreachable, thrift timeout, metastore restart, or the heartbeat thread is interrupted while a commit is in flight.

Common situations: Long-running commits exceeding the metastore's lock timeout when heartbeats fail silently; transient network partitions; metastore failover during a heavy write; heartbeat interval (lock-heartbeat-interval-ms) set longer than hive.txn.timeout.

Related errors

• Failed to heartbeat for hive lock. %s
• Hive lock heartbeat thread not active
• Failed to list all tables under namespace ${namespace}
• Failed to connect to Hive Metastore
• Failed to reconnect to Hive Metastore

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6db903a8ab87dff9. Report an issue: GitHub.