apache/pulsar · error · ReplicationException.UnavailableException

Error contacting with metadata store

Error message

Error contacting with metadata store

What it means

This UnavailableException is thrown by PulsarLedgerUnderreplicationManager.getLedgerUnreplicationInfo when the blocking metadata-store read of the under-replicated ledger node (store.get(path)) either fails (ExecutionException) or exceeds BLOCKING_CALL_TIMEOUT (TimeoutException). It signals that the BookKeeper under-replication manager could not reach or get a timely answer from the coordination service (ZooKeeper, etcd, or Oxia). The original store exception is preserved as the cause.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java:316

            if (!optRes.isPresent()) {
                log.debug().attr("ledgerId", ledgerId).log("Ledger is not marked underreplicated");
                return null;
            }

            byte[] data = optRes.get().getValue();

            UnderreplicatedLedgerFormat underreplicatedLedgerFormat = new UnderreplicatedLedgerFormat();

            underreplicatedLedgerFormat.parseFromTextFormat(data);
            PulsarUnderreplicatedLedger underreplicatedLedger = new PulsarUnderreplicatedLedger(ledgerId);
            List<String> replicaList = underreplicatedLedgerFormat.getReplicasList();
            long ctime = (underreplicatedLedgerFormat.hasCtime() ? underreplicatedLedgerFormat.getCtime()
                    : UnderreplicatedLedger.UNASSIGNED_CTIME);
            underreplicatedLedger.setCtime(ctime);
            underreplicatedLedger.setReplicaList(replicaList);
            return underreplicatedLedger;
        } catch (ExecutionException | TimeoutException ee) {
            throw new ReplicationException.UnavailableException("Error contacting with metadata store", ee);
        } catch (InterruptedException ie) {
            Thread.currentThread().interrupt();
            throw new ReplicationException.UnavailableException("Interrupted while connecting metadata store", ie);
        } catch (RuntimeException pe) {
            throw new ReplicationException.UnavailableException("Error parsing proto message", pe);
        }
    }

    @Override
    public CompletableFuture<Void> markLedgerUnderreplicatedAsync(long ledgerId, Collection<String> missingReplicas) {
        log.debug().attr("ledgerId", ledgerId).attr("missingReplicas", missingReplicas)
                .log("markLedgerUnderreplicated");
        final String path = getUrLedgerPath(ledgerId);
        final CompletableFuture<Void> createFuture = new CompletableFuture<>();
        tryMarkLedgerUnderreplicatedAsync(path, missingReplicas, createFuture);
        return createFuture;
    }

View on GitHub (pinned to 820761864e)

Solutions

1. Check metadata store connectivity from the broker (zkServer status / telnet to ZK port) and restart the Auditor once connectivity is restored.
2. Inspect the cause chain (ee.getCause()) for MetadataStoreException subtypes to identify session-expiry vs auth vs IO problems and fix accordingly (e.g. re-authenticate, fix ACLs).
3. Verify metadataServiceUri / metadata store configuration in broker.conf points to a healthy ensemble.
4. If timeouts recur under load, reduce metadata store load or scale the ensemble; the blocking timeout is not user-configurable in this path.

Example fix

// before: calling underreplicated ledger info without checking store health
UnderreplicatedLedger l = urManager.getLedgerUnreplicationInfo(ledgerId);
// after: check availability first and retry transient failures
if (!metadataStore.isAvailable()) { waitForStore(); }
try {
    UnderreplicatedLedger l = urManager.getLedgerUnreplicationInfo(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    retryWithBackoff(() -> urManager.getLedgerUnreplicationInfo(ledgerId));
}

// check metadata store health before the call
if (!metadataStore.isAvailable()) {
    throw new IllegalStateException("metadata store not reachable; check metadataServiceUri and ZK ensemble");
}

// Java: no runtime type guard; instead narrow the cause
static boolean isStoreConnectivity(Throwable t) {
    return t instanceof ReplicationException.UnavailableException
        && t.getCause() instanceof MetadataStoreException;
}

try {
    UnderreplicatedLedger l = urManager.getLedgerUnreplicationInfo(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    if (e.getCause() instanceof TimeoutException) { /* retry with backoff */ }
    else { log.error("store get failed for {}", ledgerId, e); }
}

When it happens

Trigger: Calling getLedgerUnreplicationInfo (typically via AuditorElector/underreplicatedLedger lookup) when: (1) the metadata store session is expired or connection is down, (2) store.get() completes exceptionally (e.g. NoNode handled as absent is fine, but auth failures, session loss, or IO errors surface here), or (3) the get takes longer than BLOCKING_CALL_TIMEOUT milliseconds.

Common situations: ZooKeeper ensemble unreachable or in leader election during a broker/auditor start; network partition between broker and metadata service; metadata store overloaded (latency spike exceeding the blocking timeout); misconfigured metadataServiceUri; ZooKeeper session expiry after long GC pauses.

Related errors

• Failed to get children of ${path}
• Failed to check exist ${POLICIES_READONLY_FLAG_PATH}
• Error preloading next range
• Error when get child nodes from zk
• IOException

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/e824f4595deb8b96. Report an issue: GitHub.