apache/iceberg · error · IllegalStateException

Connection to Zookeeper timed out

Error message

Connection to Zookeeper timed out

What it means

ZkLockFactory.open() starts a Curator client and waits blockUntilConnected(connectionTimeoutMs); if the ZooKeeper session is not established within the timeout, open() throws IllegalStateException with this message. The factory cannot be used without a live ZooKeeper connection.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java:126

  @Override
  public void open() {
    if (isOpen) {
      LOG.debug("ZkLockFactory already opened for lockId: {}.", lockId);
      return;
    }

    this.client =
        CuratorFrameworkFactory.builder()
            .connectString(connectString)
            .sessionTimeoutMs(sessionTimeoutMs)
            .connectionTimeoutMs(connectionTimeoutMs)
            .retryPolicy(createRetryPolicy())
            .build();
    client.start();

    try {
      if (!client.blockUntilConnected(connectionTimeoutMs, TimeUnit.MILLISECONDS)) {
        throw new IllegalStateException("Connection to Zookeeper timed out");
      }

      this.taskSharedCount = new SharedCount(client, getTaskSharePath(), 0);
      this.recoverySharedCount = new SharedCount(client, getRecoverySharedPath(), 0);
      taskSharedCount.start();
      recoverySharedCount.start();
      isOpen = true;
      LOG.info("ZkLockFactory initialized for lockId: {}.", lockId);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new RuntimeException("Interrupted while connecting to Zookeeper", e);
    } catch (Exception e) {
      closeQuietly();
      throw new RuntimeException("Failed to initialize SharedCount", e);
    }
  }

  private String getTaskSharePath() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the zooKeeperAddress and port are correct and reachable (nc/zkCli from the Flink host).
2. Increase the connectionTimeoutMs value in ZkLockFactory.builder() if the network is slow.
3. Check ZooKeeper server health/logs (ensemble quorum may be lost).
4. Fix firewall/Security Group/DNS so the Flink node can reach the quorum.
5. Retry open() once the ensemble is healthy; keep retryPolicy generous for session recovery.

Example fix

// before
ZkLockFactory.builder().setZooKeeperAddress("zk-internal:2181") // unreachable
  .setConnectionTimeoutMs(1000).build();
// after
ZkLockFactory.builder().setZooKeeperAddress("zk-1:2181,zk-2:2181,zk-3:2181")
  .setConnectionTimeoutMs(15000).build();

// from the Flink host, before opening the factory
echo ruok | nc zk-1 2181  # expect imok

try {
  factory.open();
} catch (IllegalStateException e) {
  LOG.error("ZK connect timeout: {}", e.getMessage());
  // retry open with backoff or fail the trigger deployment
}

When it happens

Trigger: Calling open() when ZooKeeper is unreachable, slow, or overloaded and connectionTimeoutMs elapses before the session is established; also wrong zooKeeperAddress/host/port or the ensemble rejecting SASL/auth.

Common situations: Wrong quorum address or port (2181) in config; ZooKeeper ensemble down or in maintenance; firewall/Security Group blocking the port; connectionTimeoutMs set too low for a cross-datacenter ZooKeeper; DNS resolving to an unreachable host.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to initialize SharedCount
• Connection to Zookeeper timed out
• Failed to initialize SharedCount
• Failed to check Zookeeper lock status
• Interrupted while connecting to Zookeeper

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/dcb73744e80b4e9c. Report an issue: GitHub.