{"record":{"id":"6db903a8ab87dff9","repo":"apache/iceberg","slug":"failed-to-heartbeat-for-lock-d","errorCode":null,"errorMessage":"Failed to heartbeat for lock: %d","messagePattern":"Failed to heartbeat for lock: (.+?)","errorType":"exception","errorClass":"CommitFailedException","httpStatus":null,"severity":"error","filePath":"hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java","lineNumber":492,"sourceCode":"\n    Heartbeat(ClientPool<IMetaStoreClient, TException> hmsClients, long lockId, long intervalMs) {\n      this.hmsClients = hmsClients;\n      this.lockId = lockId;\n      this.intervalMs = intervalMs;\n      this.future = null;\n    }\n\n    @Override\n    public void run() {\n      try {\n        hmsClients.run(\n            client -> {\n              client.heartbeat(0, lockId);\n              return null;\n            });\n      } catch (TException | InterruptedException e) {\n        this.encounteredException = e;\n        throw new CommitFailedException(e, \"Failed to heartbeat for lock: %d\", lockId);\n      }\n    }\n\n    public void schedule(ScheduledExecutorService scheduler) {\n      future =\n          scheduler.scheduleAtFixedRate(this, intervalMs / 2, intervalMs, TimeUnit.MILLISECONDS);\n    }\n\n    boolean active() {\n      return future != null && !future.isCancelled();\n    }\n\n    public void cancel() {\n      if (future != null) {\n        future.cancel(false);\n      }\n    }\n  }","sourceCodeStart":474,"sourceCodeEnd":510,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java#L474-L510","documentation":"MetastoreLock runs a scheduled heartbeat task that periodically calls the metastore heartbeat RPC to keep the acquired lock alive. If the heartbeat fails with a TException or InterruptedException, the exception is recorded and a CommitFailedException with this message is thrown, signaling the lock may have been lost and the commit must be retried. The lock will typically expire (hive.txn.timeout) without heartbeats.","triggerScenarios":"Scheduled heartbeat of an acquired table lock fails: metastore becomes unreachable, thrift timeout, metastore restart, or the heartbeat thread is interrupted while a commit is in flight.","commonSituations":"Long-running commits exceeding the metastore's lock timeout when heartbeats fail silently; transient network partitions; metastore failover during a heavy write; heartbeat interval (lock-heartbeat-interval-ms) set longer than hive.txn.timeout.","solutions":["Retry the commit; the CommitFailedException is the designed signal that the lock was lost and the commit is safe to re-attempt.","Set lock-heartbeat-interval-ms well below hive.txn.timeout so a missed heartbeat still leaves time to recover.","Fix metastore connectivity/uptime issues that interrupt heartbeats (network, metastore restarts, connection pools).","Check the recorded encounteredException cause for the underlying thrift error before tuning."],"exampleFix":"// before\ntable.properties().set(\"lock-heartbeat-interval-ms\", \"600000\"); // > txn timeout\n// after\ntable.properties().set(\"lock-heartbeat-interval-ms\", \"30000\"); // well below hive.txn.timeout","handlingStrategy":"retry","validationCode":"long heartbeatMs = Long.parseLong(table.properties().getOrDefault(\n    \"lock-heartbeat-interval-ms\", \"30000\"));\nlong txnTimeoutMs = Long.parseLong(hiveConf.get(\"hive.txn.timeout\"));\nif (heartbeatMs * 2 >= txnTimeoutMs) {\n  throw new IllegalArgumentException(\"heartbeat interval must be well below hive.txn.timeout\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  table.commit(apply);\n} catch (CommitFailedException e) {\n  if (e.getMessage().startsWith(\"Failed to heartbeat for lock\")) {\n    // lock lost due to heartbeat failure; refresh and re-attempt\n    backoffRetry(() -> { table.refresh(); table.commit(apply); });\n  } else throw e;\n}","preventionTips":["Set lock-heartbeat-interval-ms far below hive.txn.timeout","Monitor metastore uptime/network between workers and metastore","Alert on repeated CommitFailedException with heartbeat messages","Keep commit durations short to limit exposure to lock loss"],"tags":["hive","locking","heartbeat","network"],"backgroundTag":"lock-heartbeat-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}