apache/iceberg · error · IllegalStateException

Connection to Zookeeper timed out

Error message

Connection to Zookeeper timed out

What it means

Thrown by ZkLockFactory.open() when the Curator client cannot reach Zookeeper within the configured connectionTimeoutMs; blockUntilConnected returns false and the factory refuses to start. It guards the maintenance-table lock infrastructure (SharedCount-based) against operating without a live session.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/ZkLockFactory.java:126

  @Override
  public void open() {
    if (isOpen) {
      LOG.debug("ZkLockFactory already opened for lockId: {}.", lockId);
      return;
    }

    this.client =
        CuratorFrameworkFactory.builder()
            .connectString(connectString)
            .sessionTimeoutMs(sessionTimeoutMs)
            .connectionTimeoutMs(connectionTimeoutMs)
            .retryPolicy(createRetryPolicy())
            .build();
    client.start();

    try {
      if (!client.blockUntilConnected(connectionTimeoutMs, TimeUnit.MILLISECONDS)) {
        throw new IllegalStateException("Connection to Zookeeper timed out");
      }

      this.taskSharedCount = new SharedCount(client, getTaskSharePath(), 0);
      this.recoverySharedCount = new SharedCount(client, getRecoverySharedPath(), 0);
      taskSharedCount.start();
      recoverySharedCount.start();
      isOpen = true;
      LOG.info("ZkLockFactory initialized for lockId: {}.", lockId);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
      throw new RuntimeException("Interrupted while connecting to Zookeeper", e);
    } catch (Exception e) {
      closeQuietly();
      throw new RuntimeException("Failed to initialize SharedCount", e);
    }
  }

  private String getTaskSharePath() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the Zookeeper connect string and that the quorum is reachable from the Flink job (telnet/nc host:2181)
2. Increase the connection timeout in the lock configuration (e.g. lock.client.connection-timeout-ms) and retry the job
3. Check Zookeeper server logs and cluster health (zookeeper quorum, SASL/TLS settings)
4. Ensure Flink TaskManagers have network/DNS access to the Zookeeper ensemble

Example fix

// before
TableMaintenance.locks(ZookeeperLockFactory.builder()
    .withZkAddress("zk-1:2181")
    .withConnectionTimeout(5000))
// after
TableMaintenance.locks(ZookeeperLockFactory.builder()
    .withZkAddress("zk-1:2181,zk-2:2181,zk-3:2181")
    .withConnectionTimeout(60000))

// before opening the lock factory
try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(zkHost, zkPort), 5000); // throws if unreachable
}

// ZookeeperLockFactory / ZkLockFactory.open
try {
  lockFactory.open();
} catch (RuntimeException e) {
  if (e.getMessage().contains("Connection to Zookeeper timed out")) {
    // backoff and retry open, or fail the task with a clear message
  } else { throw e; }
}

When it happens

Trigger: ZkLockFactoryBuilder.build()/open() called while the Zookeeper quorum is unreachable, DNS fails, the connect string is wrong, TLS/auth is misconfigured, or the network is slow enough that connectionTimeoutMs elapses before the session is established.

Common situations: Zookeeper cluster down or being restarted; wrong connectString (host/port) in table lock properties; firewall or security group blocking the client port; Zookeeper under load causing slow session establishment; container startup ordering (Flink job starts before Zookeeper is ready).

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Connection to Zookeeper timed out
• Connection to Zookeeper timed out
• Cannot initialize JDBC table maintenance lock: Query timed o
• Connection to Zookeeper timed out
• Failed to acquire Zookeeper lock

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/941fbb1630a9c889. Report an issue: GitHub.