apache/pulsar · error · ReplicationException.UnavailableException

Failed to acuire under-replicated ledger

Error message

Failed to acuire under-replicated ledger

What it means

Thrown by acquireUnderreplicatedLedger when internalAcquireUnderreplicatedLedger fails: the attempt to create the ephemeral lock node (store.put on the UR lock path with CreateOption.Ephemeral, blocked up to BLOCKING_CALL_TIMEOUT) throws ExecutionException, times out, or is interrupted. The rereplication worker did not obtain exclusive ownership of the ledger, so it must not proceed to replicate it.

Solutions

1. Retry acquiring the ledger after a backoff — another worker may legitimately hold or the outage may be transient.
2. Verify metadata store health (session, connectivity) before resuming rereplication work.
3. Check for a stale/leaked ephemeral lock path from a crashed worker; ephemeral nodes clear on session close, so confirm the dead worker's ZK session is gone.
4. Ensure worker threads are not interrupted during shutdown before acquisition completes; sequence shutdown after in-flight acquires.

Example fix

// before: one failed acquire aborts the ledger processing
urManager.acquireUnderreplicatedLedger(ledgerId);
replicate(ledgerId);
// after: bounded retry around acquisition
for (int i = 0; i < 3; i++) {
    try {
        urManager.acquireUnderreplicatedLedger(ledgerId);
        replicate(ledgerId);
        return;
    } catch (ReplicationException.UnavailableException e) {
        Thread.sleep(1000L * (i + 1));
    }
}

// confirm the store is writable and reachable before attempting the lock
metadataStore.put(probePath, new byte[0], Optional.empty()).join();
// optionally check an existing lock holder
boolean locked = metadataStore.get(lockPath).join().isPresent();

static boolean isAcquisitionFailure(ReplicationException e) {
    return e instanceof ReplicationException.UnavailableException
        && String.valueOf(e.getMessage()).contains("acuire");
}

try {
    urManager.acquireUnderreplicatedLedger(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    if (Thread.interrupted()) return; // shutdown
    scheduleRetryWithBackoff(ledgerId); // transient: retry acquisition
}

When it happens

Trigger: A BookKeeper Auditor/rereplication worker calls acquireUnderreplicatedLedger(ledgerId) when: the metadata store is unavailable (session expired, connection lost), the ephemeral lock creation exceeds BLOCKING_CALL_TIMEOUT, or the calling thread is interrupted while waiting for the put to complete.

Common situations: Multiple rereplication workers racing during a bookie failure while the metadata store is degraded; ZooKeeper failover in progress; broker shutdown interrupting the worker mid-acquire; network latency pushing lock creation past the blocking timeout.

Related errors

• Error contacting with metadata store
• IOException
• IOException
• ${e.getMessage()}
• Error contacting metadata store

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/b9c761d935d5ae15. Report an issue: GitHub.

Appendix: source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java:418

            store.put(path, newUrLedgerData, Optional.of(optRes.get().getStat().getVersion()))
                    .thenRun(() -> {
                        FutureUtils.complete(finalFuture, null);
                    }).exceptionally(ex -> {
                        FutureUtils.completeExceptionally(finalFuture, ex);
                        return null;
                    });
        }).exceptionally(ex -> {
            FutureUtils.completeExceptionally(finalFuture, ex);
            return null;
        });
    }

    @Override
    public void acquireUnderreplicatedLedger(long ledgerId) throws ReplicationException {
        try {
            internalAcquireUnderreplicatedLedger(ledgerId);
        } catch (ExecutionException | TimeoutException | InterruptedException e) {
            throw new ReplicationException.UnavailableException("Failed to acuire under-replicated ledger", e);
        }
    }

    private void internalAcquireUnderreplicatedLedger(long ledgerId) throws ExecutionException,
            InterruptedException, TimeoutException {
        String lockPath = getUrLedgerLockPath(urLockPath, ledgerId);
        store.put(lockPath, LOCK_DATA, Optional.of(-1L), EnumSet.of(CreateOption.Ephemeral))
                .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);
    }

    @Override
    public void markLedgerReplicated(long ledgerId) throws ReplicationException.UnavailableException {
        log.debug().attr("ledgerId", ledgerId).log("markLedgerReplicated");
        try {
            Lock l = heldLocks.get(ledgerId);
            if (l != null) {
                store.delete(getUrLedgerPath(ledgerId), Optional.of(l.getLedgerNodeVersion()))
                        .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);

View on GitHub (pinned to 820761864e)