apache/iceberg · error · IllegalStateException

Connection to Zookeeper timed out

Error message

Connection to Zookeeper timed out

What it means

ZkLockFactory.open() starts a Curator ZooKeeper client and blocks until the connection is established within connectionTimeoutMs. If the connection is not established in time, an IllegalStateException with "Connection to Zookeeper timed out" is thrown. This means the maintenance trigger lock could not reach the ZooKeeper ensemble.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java:126

  @Override
  public void open() {
    if (isOpen) {
      LOG.debug("ZkLockFactory already opened for lockId: {}.", lockId);
      return;
    }

    this.client =
        CuratorFrameworkFactory.builder()
            .connectString(connectString)
            .sessionTimeoutMs(sessionTimeoutMs)
            .connectionTimeoutMs(connectionTimeoutMs)
            .retryPolicy(createRetryPolicy())
            .build();
    client.start();

    try {
      if (!client.blockUntilConnected(connectionTimeoutMs, TimeUnit.MILLISECONDS)) {
        throw new IllegalStateException("Connection to Zookeeper timed out");
      }

      this.taskSharedCount = new SharedCount(client, getTaskSharePath(), 0);
      this.recoverySharedCount = new SharedCount(client, getRecoverySharedPath(), 0);
      taskSharedCount.start();
      recoverySharedCount.start();
      isOpen = true;
      LOG.info("ZkLockFactory initialized for lockId: {}.", lockId);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new RuntimeException("Interrupted while connecting to Zookeeper", e);
    } catch (Exception e) {
      closeQuietly();
      throw new RuntimeException("Failed to initialize SharedCount", e);
    }
  }

  private String getTaskSharePath() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the ZooKeeper connect string and that the port is reachable (telnet/nc from the Flink host).
2. Increase connectionTimeoutMs in the ZkLockFactory configuration (e.g., to 30000-60000ms) for slow or TLS-enabled clusters.
3. Check ZooKeeper server health/logs; restart or fix the ensemble if quorum is lost.
4. Ensure ZooKeeper is up before the Flink job starts (init containers / readiness probes).
5. Check DNS resolution and network policies/firewalls between the Flink cluster and ZooKeeper.

Example fix

// before
ZkLockFactory.factory("zk:2181") // wrong host, short timeout
    .withConnectionTimeoutMs(1000)
    .create();
// after
ZkLockFactory.factory("zk1:2181,zk2:2181,zk3:2181")
    .withConnectionTimeoutMs(30000)
    .create();

// before creating the factory
try (Socket s = new Socket()) { s.connect(new InetSocketAddress("zk1", 2181), 5000); } // throws if unreachable

try { lockFactory.open(); } catch (IllegalStateException e) { if (e.getMessage().contains("timed out")) { /* increase timeout / check ZK */ } throw e; }

When it happens

Trigger: Calling open() when the ZooKeeper quorum is unreachable, slow, or when connectionTimeoutMs is too small for the environment (TLS handshake, large cluster, DNS slowness).

Common situations: Wrong zookeeper URI/host in configuration; firewall or security group blocking the client port (2181); ZooKeeper ensemble down or in a lossy quorum; TLS misconfiguration adding handshake latency; container start ordering (app starting before ZooKeeper is ready).

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Connection to Zookeeper timed out
• Interrupted while connecting to Zookeeper
• Failed to initialize SharedCount
• Failed to check Zookeeper lock status
• Failed to release lock

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/5e6afaa7080438e4. Report an issue: GitHub.