apache/pulsar · error · ReplicationException.UnavailableException

Error contacting metadata store

Error message

Error contacting metadata store

What it means

Thrown by markLedgerReplicated when deleting the under-replicated ledger node (store.delete with expected version) fails with an ExecutionException whose cause is neither NotFoundException nor BadVersionException (both benign), or when the delete/parent-cleanup call times out. It means the metadata store could not confirm removal of the marker, so the ledger may remain flagged as under-replicated even though replication succeeded.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java:479

                                || (ee.getCause() instanceof MetadataStoreException
                                        && ee.getCause().getCause()
                                                instanceof KeeperException.NotEmptyException);
                        if (!isNotEmpty) {
                            log.warn().exception(ee).log("Error deleting underreplicated ledger parent node");
                        }
                    }
                }
            }
        } catch (ExecutionException ee) {
            if (ee.getCause() instanceof MetadataStoreException.NotFoundException) {
                // this is ok
            } else if (ee.getCause() instanceof MetadataStoreException.BadVersionException) {
                // if this is the case, some has marked the ledger
                // for rereplication again. Leave the underreplicated
                // znode in place, so the ledger is checked.
            } else {
                log.error().exception(ee).log("Error deleting underreplicated ledger node");
                throw new ReplicationException.UnavailableException("Error contacting metadata store", ee);
            }
        } catch (TimeoutException ex) {
            throw new ReplicationException.UnavailableException("Error contacting metadata store", ex);
        } catch (InterruptedException ie) {
            Thread.currentThread().interrupt();
            throw new ReplicationException.UnavailableException("Interrupted while contacting metadata store", ie);
        } finally {
            releaseUnderreplicatedLedger(ledgerId);
        }
    }

    /**
     * Get a list of all the underreplicated ledgers which have been
     * marked for rereplication, filtered by the predicate on the replicas list.
     *
     * <p>Replicas list of an underreplicated ledger is the list of the bookies which are part of
     * the ensemble of this ledger and are currently unavailable/down.
     *

View on GitHub (pinned to 820761864e)

Solutions

1. Retry markLedgerReplicated (or verify the node's state) once the metadata store is reachable; if NotFoundException is then returned, the marker is gone and the operation effectively succeeded.
2. Check ee.getCause() for MetadataStoreException type: session/auth issues require reconnection or ACL fixes; transient IO errors warrant retry.
3. Verify write permissions (ACLs) on the under-replicated ledger path if the cause indicates permission failure.
4. Confirm the ledger actually replicated (via the admin API) and rely on the Auditor to re-check the leftover marker if the node remains.

Example fix

// before: single attempt, fails hard on transient store error
urManager.markLedgerReplicated(ledgerId);
// after: retry transient failures and confirm the marker state
try {
    urManager.markLedgerReplicated(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    if (isTransient(e.getCause())) {
        retryWithBackoff(() -> urManager.markLedgerReplicated(ledgerId));
    } else {
        throw e;
    }
}

// verify the marker still exists and matches the held lock version before deleting
var res = metadataStore.get(urLedgerPath).join();
if (res.isEmpty()) { return; /* already replicated — nothing to do */ }
long version = res.get().getStat().getVersion();

static boolean isBenignCause(ExecutionException ee) {
    return ee.getCause() instanceof MetadataStoreException.NotFoundException
        || ee.getCause() instanceof MetadataStoreException.BadVersionException;
}

try {
    urManager.markLedgerReplicated(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    Throwable c = e.getCause();
    if (isTransient(c)) { retryWithBackoff(() -> urManager.markLedgerReplicated(ledgerId)); }
    else { log.error("non-transient delete failure for {}", ledgerId, e); }
}

When it happens

Trigger: Calling markLedgerReplicated(ledgerId) after successful rereplication when: the delete fails with an unexpected MetadataStoreException (connection lost, session expired, ACL/permission denied), or the blocking get/delete exceeds BLOCKING_CALL_TIMEOUT. NotFound (already deleted) and BadVersion (someone re-marked the ledger) are intentionally not errors.

Common situations: ZooKeeper session expiry mid-cleanup after a long replication run; permission/ACL changes on the /ledgers/underreplicated subtree; metadata store outage during auditor cleanup; the ledger being re-marked by a concurrent audit while being cleaned (benign cases pass through, others surface here).

Related errors

• Error contacting with metadata store
• Error while getting ReplicationWorkerId rereplicating Ledger
• Failed to write cookie for bookie ${bookieId}
• Failed to initialize BookKeeper metadata
• Metadata store address argument is required (--metadata-stor

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/66f685d7ebad0380. Report an issue: GitHub.