{"record":{"id":"9247951c43d9dd1b","repo":"apache/iceberg","slug":"failed-to-acquire-zookeeper-lock","errorCode":null,"errorMessage":"Failed to acquire Zookeeper lock","messagePattern":"Failed to acquire Zookeeper lock","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java","lineNumber":218,"sourceCode":"    }\n\n    @Override\n    public boolean tryLock() {\n      VersionedValue<Integer> versionedValue = sharedCount.getVersionedValue();\n      if (isHeld(versionedValue)) {\n        LOG.debug(\"Lock is already held for path: {}\", lockPath);\n        return false;\n      }\n\n      try {\n        boolean acquired = sharedCount.trySetCount(versionedValue, LOCKED);\n        if (!acquired) {\n          LOG.debug(\"Failed to acquire lock for path: {}\", lockPath);\n        }\n\n        return acquired;\n      } catch (Exception e) {\n        LOG.warn(\"Failed to acquire Zookeeper lock\", e);\n        return false;\n      }\n    }\n\n    @Override\n    public boolean isHeld() {\n      return isHeld(sharedCount.getVersionedValue());\n    }\n\n    private static boolean isHeld(VersionedValue<Integer> versionedValue) {\n      try {\n        return versionedValue.getValue() == LOCKED;\n      } catch (Exception e) {\n        throw new RuntimeException(\"Failed to check Zookeeper lock status\", e);\n      }\n    }\n\n    @Override","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java#L200-L236","documentation":"ZkLockFactory.tryLock() wraps sharedCount.trySetCount(...) in a try/catch; when any exception occurs while acquiring the ZooKeeper-backed SharedCount lock, it is logged as \"Failed to acquire Zookeeper lock\" and the method returns false instead of propagating. It signals that lock acquisition failed for an infrastructure reason (connectivity, session loss, ZK errors), not merely that another job already holds the lock (which returns false without this message).","triggerScenarios":"Calling tryLock() on a ZkLock when: the Curator/ZooKeeper session has expired or is disconnected; the ZooKeeper ensemble is down or unreachable; KeeperException occurs during trySetCount; version conflict on the shared count znode due to a concurrent writer; interrupt during the ZK call.","commonSituations":"ZooKeeper quorum down or network partition between the Flink cluster and ZK; session-timeout too low so sessions expire under GC pauses; another maintenance job holds the lock on the same lockId path; ZK ACL changes blocking access to /iceberg/flink/maintenance/locks/<lockId>.","solutions":["Check ZooKeeper ensemble health and network connectivity from the Flink cluster (zkServer status, telnet to client port)","Increase sessionTimeoutMs/connectionTimeoutMs and use a retry policy with enough retries (e.g. EXPONENTIAL_BACKOFF with higher maxRetries)","Verify the lockId is unique per job+table so concurrent maintenance jobs do not contend for the same lock path","Check ZK ACLs/permissions for the path /iceberg/flink/maintenance/locks/<lockId>","Once connectivity is restored, retry the trigger; tryLock returning false is safe to retry"],"exampleFix":"// before: misdiagnosing 'false' as lock contention, job silently skips maintenance\nif (!lock.tryLock()) { return; }\n\n// after: distinguish infra failure from contention and verify ZK connectivity\nif (!lock.tryLock()) {\n  if (!client.getZookeeperClient().blockUntilConnectedOrTimedOut()) {\n    alertOps(\"ZK lock acquisition failed due to infra, not contention\");\n  }\n  return; // safe to retry on next trigger\n}","handlingStrategy":"retry","validationCode":"// before relying on tryLock, verify ZK connectivity\nboolean zkReachable = client.getZookeeperClient().blockUntilConnectedOrTimedOut();\nif (!zkReachable) {\n  alertOps(\"ZooKeeper unreachable; maintenance triggers will be skipped\");\n}","typeGuard":null,"tryCatchPattern":"// tryLock never throws; treat 'false' as contention or infra failure and retry next trigger\nif (!lock.tryLock()) {\n  LOG.warn(\"Lock not acquired; will retry on next trigger\");\n  return;\n}\ntry { doMaintenance(); } finally { lock.unlock(); }","preventionTips":["Monitor ZooKeeper ensemble health and Flink-to-ZK connectivity; alert before sessions expire","Configure generous sessionTimeoutMs/connectionTimeoutMs and a retry policy with adequate retries (EXPONENTIAL_BACKOFF)","Use a unique lockId per job+table to avoid cross-job lock contention","Watch the maintenance error-counter metric; repeated failed tryLocks indicate infra issues, not contention"],"tags":["zookeeper","distributed-lock","flink","network"],"backgroundTag":"connection-refused","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}