apache/pulsar · error · ReplicationException.UnavailableException

Failed to acuire under-replicated ledger

Error message

Failed to acuire under-replicated ledger

What it means

Thrown by acquireUnderreplicatedLedger when internalAcquireUnderreplicatedLedger fails: the attempt to create the ephemeral lock node (store.put on the UR lock path with CreateOption.Ephemeral, blocked up to BLOCKING_CALL_TIMEOUT) throws ExecutionException, times out, or is interrupted. The rereplication worker did not obtain exclusive ownership of the ledger, so it must not proceed to replicate it.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java:418

            store.put(path, newUrLedgerData, Optional.of(optRes.get().getStat().getVersion()))
                    .thenRun(() -> {
                        FutureUtils.complete(finalFuture, null);
                    }).exceptionally(ex -> {
                        FutureUtils.completeExceptionally(finalFuture, ex);
                        return null;
                    });
        }).exceptionally(ex -> {
            FutureUtils.completeExceptionally(finalFuture, ex);
            return null;
        });
    }

    @Override
    public void acquireUnderreplicatedLedger(long ledgerId) throws ReplicationException {
        try {
            internalAcquireUnderreplicatedLedger(ledgerId);
        } catch (ExecutionException | TimeoutException | InterruptedException e) {
            throw new ReplicationException.UnavailableException("Failed to acuire under-replicated ledger", e);
        }
    }

    private void internalAcquireUnderreplicatedLedger(long ledgerId) throws ExecutionException,
            InterruptedException, TimeoutException {
        String lockPath = getUrLedgerLockPath(urLockPath, ledgerId);
        store.put(lockPath, LOCK_DATA, Optional.of(-1L), EnumSet.of(CreateOption.Ephemeral))
                .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);
    }

    @Override
    public void markLedgerReplicated(long ledgerId) throws ReplicationException.UnavailableException {
        log.debug().attr("ledgerId", ledgerId).log("markLedgerReplicated");
        try {
            Lock l = heldLocks.get(ledgerId);
            if (l != null) {
                store.delete(getUrLedgerPath(ledgerId), Optional.of(l.getLedgerNodeVersion()))
                        .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);

View on GitHub (pinned to 820761864e)

Solutions

1. Retry acquiring the ledger after a backoff — another worker may legitimately hold or the outage may be transient.
2. Verify metadata store health (session, connectivity) before resuming rereplication work.
3. Check for a stale/leaked ephemeral lock path from a crashed worker; ephemeral nodes clear on session close, so confirm the dead worker's ZK session is gone.
4. Ensure worker threads are not interrupted during shutdown before acquisition completes; sequence shutdown after in-flight acquires.

Example fix

// before: one failed acquire aborts the ledger processing
urManager.acquireUnderreplicatedLedger(ledgerId);
replicate(ledgerId);
// after: bounded retry around acquisition
for (int i = 0; i < 3; i++) {
    try {
        urManager.acquireUnderreplicatedLedger(ledgerId);
        replicate(ledgerId);
        return;
    } catch (ReplicationException.UnavailableException e) {
        Thread.sleep(1000L * (i + 1));
    }
}

// confirm the store is writable and reachable before attempting the lock
metadataStore.put(probePath, new byte[0], Optional.empty()).join();
// optionally check an existing lock holder
boolean locked = metadataStore.get(lockPath).join().isPresent();

static boolean isAcquisitionFailure(ReplicationException e) {
    return e instanceof ReplicationException.UnavailableException
        && String.valueOf(e.getMessage()).contains("acuire");
}

try {
    urManager.acquireUnderreplicatedLedger(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    if (Thread.interrupted()) return; // shutdown
    scheduleRetryWithBackoff(ledgerId); // transient: retry acquisition
}

When it happens

Trigger: A BookKeeper Auditor/rereplication worker calls acquireUnderreplicatedLedger(ledgerId) when: the metadata store is unavailable (session expired, connection lost), the ephemeral lock creation exceeds BLOCKING_CALL_TIMEOUT, or the calling thread is interrupted while waiting for the put to complete.

Common situations: Multiple rereplication workers racing during a bookie failure while the metadata store is degraded; ZooKeeper failover in progress; broker shutdown interrupting the worker mid-acquire; network latency pushing lock creation past the blocking timeout.

Related errors

• IOException
• IOException
• Error contacting with metadata store
• Timeout during managed ledger close
• Timeout during managed ledger delete operation

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/b9c761d935d5ae15. Report an issue: GitHub.