apache/iceberg · warning

Failed to acquire Zookeeper lock

Error message

Failed to acquire Zookeeper lock

What it means

ZkLockFactory.tryLock() wraps sharedCount.trySetCount(...) in a try/catch; when any exception occurs while acquiring the ZooKeeper-backed SharedCount lock, it is logged as "Failed to acquire Zookeeper lock" and the method returns false instead of propagating. It signals that lock acquisition failed for an infrastructure reason (connectivity, session loss, ZK errors), not merely that another job already holds the lock (which returns false without this message).

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java:218

    }

    @Override
    public boolean tryLock() {
      VersionedValue<Integer> versionedValue = sharedCount.getVersionedValue();
      if (isHeld(versionedValue)) {
        LOG.debug("Lock is already held for path: {}", lockPath);
        return false;
      }

      try {
        boolean acquired = sharedCount.trySetCount(versionedValue, LOCKED);
        if (!acquired) {
          LOG.debug("Failed to acquire lock for path: {}", lockPath);
        }

        return acquired;
      } catch (Exception e) {
        LOG.warn("Failed to acquire Zookeeper lock", e);
        return false;
      }
    }

    @Override
    public boolean isHeld() {
      return isHeld(sharedCount.getVersionedValue());
    }

    private static boolean isHeld(VersionedValue<Integer> versionedValue) {
      try {
        return versionedValue.getValue() == LOCKED;
      } catch (Exception e) {
        throw new RuntimeException("Failed to check Zookeeper lock status", e);
      }
    }

    @Override

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check ZooKeeper ensemble health and network connectivity from the Flink cluster (zkServer status, telnet to client port)
2. Increase sessionTimeoutMs/connectionTimeoutMs and use a retry policy with enough retries (e.g. EXPONENTIAL_BACKOFF with higher maxRetries)
3. Verify the lockId is unique per job+table so concurrent maintenance jobs do not contend for the same lock path
4. Check ZK ACLs/permissions for the path /iceberg/flink/maintenance/locks/<lockId>
5. Once connectivity is restored, retry the trigger; tryLock returning false is safe to retry

Example fix

// before: misdiagnosing 'false' as lock contention, job silently skips maintenance
if (!lock.tryLock()) { return; }

// after: distinguish infra failure from contention and verify ZK connectivity
if (!lock.tryLock()) {
  if (!client.getZookeeperClient().blockUntilConnectedOrTimedOut()) {
    alertOps("ZK lock acquisition failed due to infra, not contention");
  }
  return; // safe to retry on next trigger
}

// before relying on tryLock, verify ZK connectivity
boolean zkReachable = client.getZookeeperClient().blockUntilConnectedOrTimedOut();
if (!zkReachable) {
  alertOps("ZooKeeper unreachable; maintenance triggers will be skipped");
}

// tryLock never throws; treat 'false' as contention or infra failure and retry next trigger
if (!lock.tryLock()) {
  LOG.warn("Lock not acquired; will retry on next trigger");
  return;
}
try { doMaintenance(); } finally { lock.unlock(); }

When it happens

Trigger: Calling tryLock() on a ZkLock when: the Curator/ZooKeeper session has expired or is disconnected; the ZooKeeper ensemble is down or unreachable; KeeperException occurs during trySetCount; version conflict on the shared count znode due to a concurrent writer; interrupt during the ZK call.

Common situations: ZooKeeper quorum down or network partition between the Flink cluster and ZK; session-timeout too low so sessions expire under GC pauses; another maintenance job holds the lock on the same lockId path; ZK ACL changes blocking access to /iceberg/flink/maintenance/locks/<lockId>.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• Connection to Zookeeper timed out
• Failed to release lock
• Connection to Zookeeper timed out
• Failed to release lock for path: {}
• Failed to acquire Zookeeper lock

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/9247951c43d9dd1b. Report an issue: GitHub.