{"record":{"id":"5e6afaa7080438e4","repo":"apache/iceberg","slug":"connection-to-zookeeper-timed-out-5e6afa","errorCode":null,"errorMessage":"Connection to Zookeeper timed out","messagePattern":"Connection to Zookeeper timed out","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java","lineNumber":126,"sourceCode":"  @Override\n  public void open() {\n    if (isOpen) {\n      LOG.debug(\"ZkLockFactory already opened for lockId: {}.\", lockId);\n      return;\n    }\n\n    this.client =\n        CuratorFrameworkFactory.builder()\n            .connectString(connectString)\n            .sessionTimeoutMs(sessionTimeoutMs)\n            .connectionTimeoutMs(connectionTimeoutMs)\n            .retryPolicy(createRetryPolicy())\n            .build();\n    client.start();\n\n    try {\n      if (!client.blockUntilConnected(connectionTimeoutMs, TimeUnit.MILLISECONDS)) {\n        throw new IllegalStateException(\"Connection to Zookeeper timed out\");\n      }\n\n      this.taskSharedCount = new SharedCount(client, getTaskSharePath(), 0);\n      this.recoverySharedCount = new SharedCount(client, getRecoverySharedPath(), 0);\n      taskSharedCount.start();\n      recoverySharedCount.start();\n      isOpen = true;\n      LOG.info(\"ZkLockFactory initialized for lockId: {}.\", lockId);\n    } catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      throw new RuntimeException(\"Interrupted while connecting to Zookeeper\", e);\n    } catch (Exception e) {\n      closeQuietly();\n      throw new RuntimeException(\"Failed to initialize SharedCount\", e);\n    }\n  }\n\n  private String getTaskSharePath() {","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java#L108-L144","documentation":"ZkLockFactory.open() starts a Curator ZooKeeper client and blocks until the connection is established within connectionTimeoutMs. If the connection is not established in time, an IllegalStateException with \"Connection to Zookeeper timed out\" is thrown. This means the maintenance trigger lock could not reach the ZooKeeper ensemble.","triggerScenarios":"Calling open() when the ZooKeeper quorum is unreachable, slow, or when connectionTimeoutMs is too small for the environment (TLS handshake, large cluster, DNS slowness).","commonSituations":"Wrong zookeeper URI/host in configuration; firewall or security group blocking the client port (2181); ZooKeeper ensemble down or in a lossy quorum; TLS misconfiguration adding handshake latency; container start ordering (app starting before ZooKeeper is ready).","solutions":["Verify the ZooKeeper connect string and that the port is reachable (telnet/nc from the Flink host).","Increase connectionTimeoutMs in the ZkLockFactory configuration (e.g., to 30000-60000ms) for slow or TLS-enabled clusters.","Check ZooKeeper server health/logs; restart or fix the ensemble if quorum is lost.","Ensure ZooKeeper is up before the Flink job starts (init containers / readiness probes).","Check DNS resolution and network policies/firewalls between the Flink cluster and ZooKeeper."],"exampleFix":"// before\nZkLockFactory.factory(\"zk:2181\") // wrong host, short timeout\n    .withConnectionTimeoutMs(1000)\n    .create();\n// after\nZkLockFactory.factory(\"zk1:2181,zk2:2181,zk3:2181\")\n    .withConnectionTimeoutMs(30000)\n    .create();","handlingStrategy":"validation","validationCode":"// before creating the factory\ntry (Socket s = new Socket()) { s.connect(new InetSocketAddress(\"zk1\", 2181), 5000); } // throws if unreachable","typeGuard":null,"tryCatchPattern":"try { lockFactory.open(); } catch (IllegalStateException e) { if (e.getMessage().contains(\"timed out\")) { /* increase timeout / check ZK */ } throw e; }","preventionTips":["Use the full ensemble connect string (all ZK hosts), not a single host.","Set connectionTimeoutMs generously (30s+) especially with TLS.","Ensure ZooKeeper readiness before starting the Flink job (init containers/probes).","Test port reachability and DNS from the Flink nodes."],"tags":["zookeeper","timeout","connection","flink","lock"],"backgroundTag":"connection-timeout","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}