apache/pulsar · error · ReplicationException.UnavailableException

Error while parsing ZK data of lock

Error message

Error while parsing ZK data of lock

What it means

getReplicationWorkerIdRereplicatingLedger(long) parses the lock znode payload with LockDataFormat.parseFromTextFormat(). Any RuntimeException during parsing (malformed/unexpected bytes, legacy or hand-edited znode data, format drift between BookKeeper versions) is wrapped as ReplicationException.UnavailableException('Error while parsing ZK data of lock'). Unlike the other cases here, ZooKeeper is healthy — the stored lock data itself is unreadable.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java:912

                // this is ok.
                return null;
            }

            byte[] lockData = optRes.get().getValue();
            LockDataFormat lock = new LockDataFormat();
            lock.parseFromTextFormat(lockData);
            return lock.getBookieId();
        } catch (ExecutionException | TimeoutException e) {
            log.error().exception(e).log("Error while getting ReplicationWorkerId rereplicating Ledger");
            throw new ReplicationException.UnavailableException(
                    "Error while getting ReplicationWorkerId rereplicating Ledger", e);
        } catch (InterruptedException e) {
            log.error().exception(e).log("Got interrupted while getting ReplicationWorkerId rereplicating Ledger");
            Thread.currentThread().interrupt();
            throw new ReplicationException.UnavailableException("Interrupted while contacting zookeeper", e);
        } catch (RuntimeException e) {
            log.error().exception(e).log("Error while parsing ZK data of lock");
            throw new ReplicationException.UnavailableException("Error while parsing ZK data of lock", e);
        }
    }

    @Override
    public void setCheckAllLedgersCTime(long checkAllLedgersCTime) throws ReplicationException.UnavailableException {
        log.debug("setCheckAllLedgersCTime");
        try {
            CheckAllLedgersFormat builder = new CheckAllLedgersFormat();
            builder.setCheckAllLedgersCTime(checkAllLedgersCTime);
            byte[] checkAllLedgersFormatByteArray = builder.toByteArray();

            store.put(checkAllLedgersCtimePath, checkAllLedgersFormatByteArray, Optional.empty())
                    .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);
        } catch (ExecutionException | TimeoutException ee) {
            throw new ReplicationException.UnavailableException("Error contacting zookeeper", ee);
        } catch (InterruptedException ie) {
            Thread.currentThread().interrupt();
            throw new ReplicationException.UnavailableException("Interrupted while contacting zookeeper", ie);

View on GitHub (pinned to 820761864e)

Solutions

1. Dump the znode content (zkCli get <urLockPath>/<ledgerId>) and compare against the LockDataFormat schema for your BookKeeper version
2. Delete the stale/corrupt lock znode so the ledger re-enters the underreplication flow and a worker re-acquires it cleanly
3. After version upgrades, verify lock format compatibility before resuming the ReplicationWorker
4. If frequent, capture the raw payload in the log (the cause carries the parse exception) and report a format mismatch to the BookKeeper/Pulsar project

Example fix

// before
String owner = urManager.getReplicationWorkerIdRereplicatingLedger(ledgerId); // dies on corrupt payload

// after
String owner;
try {
    owner = urManager.getReplicationWorkerIdRereplicatingLedger(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    if (e.getCause() instanceof RuntimeException) {
        log.warn("corrupt lock data for ledger {} — releasing lock", ledgerId, e);
        urManager.releaseUnderreplicatedLedger(ledgerId);
        owner = null;
    } else {
        throw e;
    }
}

// inspect the lock payload before relying on it
try {
    String owner = urManager.getReplicationWorkerIdRereplicatingLedger(ledgerId);
    if (owner == null || owner.isEmpty()) {
        log.warn("ledger {} lock has no bookie id; releasing", ledgerId);
        urManager.releaseUnderreplicatedLedger(ledgerId);
    }
} catch (ReplicationException.UnavailableException e) {
    if (e.getCause() instanceof RuntimeException) {
        log.warn("corrupt lock payload for ledger {}", ledgerId, e);
        urManager.releaseUnderreplicatedLedger(ledgerId);
    } else {
        throw e;
    }
}

static boolean isParseCorruption(ReplicationException.UnavailableException e) {
    return e.getCause() instanceof RuntimeException
        && !(e.getCause() instanceof IllegalStateException);
}

try {
    urManager.getReplicationWorkerIdRereplicatingLedger(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    if (isParseCorruption(e)) {
        urManager.releaseUnderreplicatedLedger(ledgerId); // let a worker re-lock cleanly
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling getReplicationWorkerIdRereplicatingLedger(ledgerId) when the lock znode's value is not valid LockDataFormat text (corrupt entry, data written by an incompatible BookKeeper version, manual znode edit, NPE on empty bookieId field).

Common situations: Upgrading BookKeeper between lock-data format revisions leaving stale locks; a crashed ReplicationWorker leaving a truncated/partial lock payload; operators manually deleting/rewriting underreplication znodes during incident response.

Related errors

• Error while getting ReplicationWorkerId rereplicating Ledger
• Error while parsing ZK protobuf binary data
• Timeout during update managedLedger's properties
• Failed to initialize BookKeeper metadata
• Metadata store address argument is required (--metadata-stor

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/c23ff36d962c4cef. Report an issue: GitHub.