apache/iceberg · error · RuntimeException

Failed to release lock

Error message

Failed to release lock

What it means

ZkLockFactory's unlock() wraps any exception from resetting the ZooKeeper shared count (SharedCount.setCount(UNLOCKED)) into a plain RuntimeException with the message 'Failed to release lock'. This fires when the ZK session is broken (expired session, connection loss, node deleted) so the distributed lock count cannot be reset. The lock may still be held or the ephemeral node may already be gone, so the caller cannot assume the lock was cleanly released.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java:243

      return isHeld(sharedCount.getVersionedValue());
    }

    private static boolean isHeld(VersionedValue<Integer> versionedValue) {
      try {
        return versionedValue.getValue() == LOCKED;
      } catch (Exception e) {
        throw new RuntimeException("Failed to check Zookeeper lock status", e);
      }
    }

    @Override
    public void unlock() {
      try {
        sharedCount.setCount(UNLOCKED);
        LOG.debug("Released lock for path: {}", lockPath);
      } catch (Exception e) {
        LOG.warn("Failed to release lock for path: {}", lockPath, e);
        throw new RuntimeException("Failed to release lock", e);
      }
    }
  }

  @VisibleForTesting
  RetryPolicy createRetryPolicy() {
    ZKRetryPolicies effectivePolicy =
        (retryPolicy == null) ? ZKRetryPolicies.EXPONENTIAL_BACKOFF : retryPolicy;

    switch (effectivePolicy) {
      case ONE_TIME:
        return new RetryOneTime(baseSleepTimeMs);

      case N_TIME:
        return new RetryNTimes(maxRetries, baseSleepTimeMs);

      case BOUNDED_EXPONENTIAL_BACKOFF:
        return new BoundedExponentialBackoffRetry(baseSleepTimeMs, maxSleepTimeMs, maxRetries);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check ZooKeeper client health and session timeout settings; increase sessionTimeoutMs if maintenance tasks run long.
2. Verify network connectivity and ZK ensemble stability between the Flink JobManager/TaskManagers and the ensemble.
3. Ensure the lock path (lockPath) still exists and the service account has write ACLs on it.
4. Retry the maintenance cycle; the failure is typically transient once ZK connectivity is restored.
5. Inspect the wrapped cause (getCause()) for the real Curator exception (ConnectionLossException, SessionExpiredException).

Example fix

// before
zkLockFactoryBuilder.lockPath("/iceberg/maintenance/lock").build();
// after
zkLockFactoryBuilder
    .lockPath("/iceberg/maintenance/lock")
    .connectionTimeoutMs(15000)
    .sessionTimeoutMs(120000) // tolerate long maintenance cycles
    .build();

// verify ZK connectivity before unlocking
if (!curatorClient.getZookeeperClient().isConnected()) {
  LOG.warn("ZK session down; lock release may fail");
}

try {
  lockFactory.withLock(() -> maintenance.run());
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to release lock")) {
    LOG.warn("Lock release failed; ZK session issue", e.getCause());
    // do not assume lock is free; wait for session expiry before retrying
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling unlock() (directly or via the maintenance TableMaintenance executor) when the Curator/ZooKeeper client has lost its session or the SharedCount cannot write the new count value to ZK.

Common situations: ZooKeeper session timeout during a long maintenance task; ZK ensemble restart or network partition; the lock znode/counter was deleted by session expiry while the job was still running.

Related errors

• Failed to acquire Zookeeper lock
• Failed to acquire Zookeeper lock
• Connection to Zookeeper timed out
• Interrupted while connecting to Zookeeper
• Failed to initialize SharedCount

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/c7618657727dfafe. Report an issue: GitHub.