t8y2/dbx · error · IllegalStateException

ZooKeeper path /brokers/ids does not exist

Error message

ZooKeeper path /brokers/ids does not exist

What it means

Thrown by discoverBootstrapServers after a successful ZooKeeper connection when the ensemble does not contain the /brokers/ids znode. This means the connected ZooKeeper ensemble is not managing a Kafka cluster (wrong ensemble, or Kafka stores its metadata elsewhere), so no brokers can be discovered.

Source

Thrown at agents/drivers/kafka/src/main/java/com/dbx/agent/kafka/KafkaAgent.java:356

        int connectionTimeout = intOrDefault(
            conn,
            "zookeeper_connection_timeout_ms",
            DEFAULT_ZOOKEEPER_CONNECTION_TIMEOUT_MS
        );
        CountDownLatch connected = new CountDownLatch(1);
        ZooKeeper zooKeeper = new ZooKeeper(connectString, sessionTimeout, event -> {
            if (event.getState() == Watcher.Event.KeeperState.SyncConnected) connected.countDown();
        }, zooKeeperClientConfig(conn));
        try {
            if (!connected.await(connectionTimeout, TimeUnit.MILLISECONDS)) {
                throw new IllegalStateException("Timed out connecting to ZooKeeper for Kafka broker discovery");
            }

            List<String> brokerIds;
            try {
                brokerIds = new ArrayList<>(zooKeeper.getChildren("/brokers/ids", false));
            } catch (KeeperException.NoNodeException e) {
                throw new IllegalStateException("ZooKeeper path /brokers/ids does not exist", e);
            }
            brokerIds.sort(KafkaAgent::compareBrokerIds);

            List<JsonObject> registrations = new ArrayList<>();
            for (String brokerId : brokerIds) {
                try {
                    byte[] data = zooKeeper.getData("/brokers/ids/" + brokerId, false, null);
                    registrations.add(JsonParser.parseString(new String(data, StandardCharsets.UTF_8)).getAsJsonObject());
                } catch (KeeperException.NoNodeException e) {
                    // Expected race: a broker may refresh its ephemeral node between list and read.
                    logger().debug("Kafka broker {} disappeared during ZooKeeper discovery", brokerId);
                } catch (RuntimeException e) {
                    logger().warn("Skipping malformed ZooKeeper registration for Kafka broker {}", brokerId, e);
                }
            }
            return brokerEndpoints(registrations, securityProtocol);
        } finally {
            zooKeeper.close();

View on GitHub (pinned to c0390bff16)

Solutions

1. Confirm zookeeper_connect_string points to the ZooKeeper ensemble used by the Kafka cluster, not a different one
2. Verify the Kafka cluster is running and has registered brokers under /brokers/ids (zkCli: ls /brokers/ids)
3. Bypass discovery by setting bootstrap_servers directly

When it happens

Trigger: Thrown at agents/drivers/kafka/src/main/java/com/dbx/agent/kafka/KafkaAgent.java:356 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/1f2589b21fc396f3. Report an issue: GitHub.