apache/iceberg · error · IllegalStateException

Connection to Zookeeper timed out

Error message

Connection to Zookeeper timed out

What it means

ZkLockFactory.open() starts the Curator client and blocks until connected within connectionTimeoutMs. If the connection is not established in time, it throws IllegalStateException "Connection to Zookeeper timed out", meaning the table maintenance lock cannot be used without a working ZooKeeper.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java:126

  @Override
  public void open() {
    if (isOpen) {
      LOG.debug("ZkLockFactory already opened for lockId: {}.", lockId);
      return;
    }

    this.client =
        CuratorFrameworkFactory.builder()
            .connectString(connectString)
            .sessionTimeoutMs(sessionTimeoutMs)
            .connectionTimeoutMs(connectionTimeoutMs)
            .retryPolicy(createRetryPolicy())
            .build();
    client.start();

    try {
      if (!client.blockUntilConnected(connectionTimeoutMs, TimeUnit.MILLISECONDS)) {
        throw new IllegalStateException("Connection to Zookeeper timed out");
      }

      this.taskSharedCount = new SharedCount(client, getTaskSharePath(), 0);
      this.recoverySharedCount = new SharedCount(client, getRecoverySharedPath(), 0);
      taskSharedCount.start();
      recoverySharedCount.start();
      isOpen = true;
      LOG.info("ZkLockFactory initialized for lockId: {}.", lockId);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new RuntimeException("Interrupted while connecting to Zookeeper", e);
    } catch (Exception e) {
      closeQuietly();
      throw new RuntimeException("Failed to initialize SharedCount", e);
    }
  }

  private String getTaskSharePath() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the ZooKeeper connect string and that the ensemble is reachable (zkCli smoke test)
2. Increase the connection timeout configuration to cover slow cluster startup
3. Check firewall, DNS, and Kerberos/SASL settings between Flink and ZooKeeper
4. Ensure ZooKeeper quorum health (majority of servers up) before launching the job

Example fix

// before
ZkLockFactory.builder().setConnectString("zk1:2181").setConnectionTimeoutMs(1000)...
// after
ZkLockFactory.builder().setConnectString("zk1:2181,zk2:2181,zk3:2181").setConnectionTimeoutMs(60000)...

// before opening the lock factory
Process p = Runtime.getRuntime().exec("echo ruok | nc zk1 2181");
boolean healthy = new String(p.getInputStream().readAllBytes()).trim().equals("imok");

try { factory.open(); } catch (IllegalStateException e) { /* retry with backoff or fail fast: ZK unreachable */ }

When it happens

Trigger: Creating a ZkLockFactory via TriggerLockFactory lock config where the ZooKeeper quorum is unreachable or slower than connectionTimeoutMs during open()/tryLock initialization.

Common situations: Wrong zookeeper connect string or port; ZooKeeper ensemble down or in a network partition; firewall/DNS blocking; connectionTimeoutMs too small for a slow cluster; auth/SASL handshake hanging.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Connection to Zookeeper timed out
• Connection to Zookeeper timed out
• Cannot initialize JDBC table maintenance lock: Query timed o
• Connection to Zookeeper timed out
• Failed to acquire Zookeeper lock

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/09eda428a6a74365. Report an issue: GitHub.