apache/pulsar · error · IOException

Error when get child nodes from zk

Error message

Error when get child nodes from zk

What it means

IOException thrown by LegacyHierarchicalLedgerRangeIterator.getLedgerRangeByLevel when the sync+getChildrenFromStore call to the metadata store fails with ExecutionException or TimeoutException while fetching the child nodes of a hash node in the hierarchical ledger layout. It means the ZK-style metadata store could not be read (failure or too slow).

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/LegacyHierarchicalLedgerRangeIterator.java:171

     * @param level1
     *          1st level node name
     * @param level2
     *          2nd level node name
     * @throws IOException
     */
    LedgerManager.LedgerRange getLedgerRangeByLevel(final String level1, final String level2)
            throws IOException {
        StringBuilder nodeBuilder = threadLocalNodeBuilder.get();
        nodeBuilder.setLength(0);
        nodeBuilder.append(ledgersRoot).append("/")
                .append(level1).append("/").append(level2);
        String nodePath = nodeBuilder.toString();
        List<String> ledgerNodes = null;
        try {
            ledgerNodes = store.sync(nodePath).thenCompose(__ -> store.getChildrenFromStore(nodePath))
                    .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);
        } catch (ExecutionException | TimeoutException e) {
            throw new IOException("Error when get child nodes from zk", e);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new IOException("Error when get child nodes from zk", e);
        }
        NavigableSet<Long> zkActiveLedgers =
                HierarchicalLedgerUtils.ledgerListToSet(ledgerNodes, ledgersRoot, nodePath);
        log.debug().attr("hashNode", level1 + "/" + level2)
                .attr("ledgers", zkActiveLedgers)
                .log("All active ledgers from ZK for hash node");

        return new LedgerManager.LedgerRange(zkActiveLedgers.subSet(getStartLedgerIdByLevel(level1, level2), true,
                getEndLedgerIdByLevel(level1, level2), true));
    }

    /**
     * Get the smallest cache id in a specified node /level1/level2.
     *
     * @param level1

View on GitHub (pinned to 820761864e)

Solutions

1. Check ZooKeeper/metadata-store health and network connectivity
2. Increase BLOCKING_CALL_TIMEOUT for large hierarchies
3. Inspect the wrapped cause (e.getCause()) to distinguish store errors from timeouts
4. Retry enumeration once the store is healthy

Example fix

// before
ledgerNodes = store.sync(nodePath).thenCompose(__ -> store.getChildrenFromStore(nodePath))
        .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS); // IOException on failure
// after
try {
    ledgerNodes = store.sync(nodePath).thenCompose(__ -> store.getChildrenFromStore(nodePath))
            .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);
} catch (ExecutionException e) {
    LOG.error("Failed to read children of {} from metadata store", nodePath, e.getCause());
    throw new IOException("Error when get child nodes from zk", e);
}

store.sync(ledgersRoot).orTimeout(5, TimeUnit.SECONDS).join(); // verify store reachable first

try {
    it.forEachRemaining(this::process);
} catch (IOException e) {
    if (e.getMessage().equals("Error when get child nodes from zk")) {
        LOG.error("Metadata store read failed for ledger hierarchy", e.getCause()); // retry/backoff
    }
    throw e;
}

When it happens

Trigger: preload() -> getLedgerRangeByLevel for an L1/L2 node; the underlying store future completes exceptionally or exceeds BLOCKING_CALL_TIMEOUT.

Common situations: ZooKeeper session expiry or quorum loss; network timeouts; very large child-node lists exceeding the store's response limits or the blocking timeout.

Related errors

• Failed to get children of ${path}
• Failed to check exist ${POLICIES_READONLY_FLAG_PATH}
• Error preloading next range
• Error contacting with metadata store
• Error reading list

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/04d5f537a70c34c9. Report an issue: GitHub.