apache/iceberg · warning · RuntimeException

Cannot heartbeat to a deleted lock

Error message

Cannot heartbeat to a deleted lock 

What it means

The InMemoryLockManager heartbeat task re-reads the lock content and uses LOCKS.replace(entityId, lastContent, newContent); if the lock was removed concurrently, lastContent may be null and replace() throws NullPointerException, which is rethrown as RuntimeException 'Cannot heartbeat to a deleted lock <entityId>'. It detects heartbeat racing with lock release/unlock.

Solutions

1. Ensure the lock is kept alive (heartbeated) for as long as it is used, and release it only after closing the operation
2. Shut down / cancel the heartbeat future before removing the lock entry
3. Upgrade Iceberg — recent versions guard against this heartbeat/unlock race
4. Catch the RuntimeException and treat it as lock loss; re-acquire before continuing

Example fix

// before
locks.remove(entityId); // heartbeat may still run
// after
heartbeatFuture.cancel(false);
locks.remove(entityId);

try { lockManager.acquire(entityId, ownerId); } catch (RuntimeException e) { if (e.getMessage().startsWith("Cannot heartbeat to a deleted lock")) { lockManager.acquire(entityId, ownerId); } else throw e; }

When it happens

Trigger: The heartbeat executor fires after the lock entry has been removed from the LOCKS map — e.g., lock released or expired and cleaned up while the scheduled heartbeat is still running.

Common situations: Close/shutdown racing with heartbeats; unlock followed immediately by JVM exit while the scheduled executor is mid-tick; very short heartbeat intervals amplifying the race window.

Related errors

• Fail to acquire lock
• Failed to heartbeat for hive lock while committing changes…
• Interrupted during tryLock
• Lock for currently held by , expiration
• Table already exists

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/a9956f2e88cc3465. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/util/LockManagers.java:225

      if (succeed) {
        // cleanup old heartbeat
        if (HEARTBEATS.containsKey(entityId)) {
          HEARTBEATS.remove(entityId).cancel(false);
        }

        HEARTBEATS.put(
            entityId,
            scheduler()
                .scheduleAtFixedRate(
                    () -> {
                      InMemoryLockContent lastContent = LOCKS.get(entityId);
                      try {
                        long newExpiration = System.currentTimeMillis() + heartbeatTimeoutMs();
                        LOCKS.replace(
                            entityId, lastContent, new InMemoryLockContent(ownerId, newExpiration));
                      } catch (NullPointerException e) {
                        throw new RuntimeException(
                            "Cannot heartbeat to a deleted lock " + entityId, e);
                      }
                    },
                    0,
                    heartbeatIntervalMs(),
                    TimeUnit.MILLISECONDS));

      } else {
        throw new IllegalStateException("Unable to acquire lock " + entityId);
      }
    }

    @Override
    public boolean acquire(String entityId, String ownerId) {
      try {
        Tasks.foreach(entityId)
            .retry(Integer.MAX_VALUE - 1)
            .onlyRetryOn(IllegalStateException.class)

View on GitHub (pinned to 86d9c8fc54)