apache/pulsar · error · ReplicationException.UnavailableException

Error while parsing ZK protobuf binary data

Error message

Error while parsing ZK protobuf binary data

What it means

Thrown by PulsarLedgerUnderreplicationManager.getCheckAllLedgersCTime when the bytes stored under the checkAllLedgers ctime znode cannot be parsed as a CheckAllLedgersFormat protobuf (a RuntimeException from parseFrom). It indicates corrupted or unexpected data at the well-known path, not a connectivity problem. Wrapped as ReplicationException.UnavailableException.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java:954

        log.debug("getCheckAllLedgersCTime");
        try {
            Optional<GetResult> optRes = store.get(checkAllLedgersCtimePath).get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);
            if (!optRes.isPresent()) {
                log.warn("checkAllLedgersCtimeZnode is not yet available");
                return -1;
            }
            byte[] data = optRes.get().getValue();
            CheckAllLedgersFormat checkAllLedgersFormat = new CheckAllLedgersFormat();
            checkAllLedgersFormat.parseFrom(data);
            return checkAllLedgersFormat.hasCheckAllLedgersCTime() ? checkAllLedgersFormat.getCheckAllLedgersCTime()
                    : -1;
        } catch (ExecutionException | TimeoutException ee) {
            throw new ReplicationException.UnavailableException("Error contacting zookeeper", ee);
        } catch (InterruptedException ie) {
            Thread.currentThread().interrupt();
            throw new ReplicationException.UnavailableException("Interrupted while contacting zookeeper", ie);
        } catch (RuntimeException ipbe) {
            throw new ReplicationException.UnavailableException("Error while parsing ZK protobuf binary data", ipbe);
        }
    }

    @Override
    public void setPlacementPolicyCheckCTime(long placementPolicyCheckCTime) throws
            ReplicationException.UnavailableException {
        log.debug("setPlacementPolicyCheckCTime");
        try {
            PlacementPolicyCheckFormat builder = new PlacementPolicyCheckFormat();
            builder.setPlacementPolicyCheckCTime(placementPolicyCheckCTime);
            byte[] placementPolicyCheckFormatByteArray = builder.toByteArray();
            store.put(placementPolicyCheckCtimePath, placementPolicyCheckFormatByteArray, Optional.empty())
                    .get(BLOCKING_CALL_TIMEOUT, MILLISECONDS);
        } catch (ExecutionException | TimeoutException ke) {
            throw new ReplicationException.UnavailableException("Error contacting zookeeper", ke);
        } catch (InterruptedException ie) {
            Thread.currentThread().interrupt();
            throw new ReplicationException.UnavailableException("Interrupted while contacting zookeeper", ie);

View on GitHub (pinned to 820761864e)

Solutions

1. Inspect the znode data (e.g. zookeeper-cli / getPersistedData) and confirm it is a serialized CheckAllLedgersFormat.
2. Delete the corrupted checkAllLedgers ctime znode; the code treats a missing node as -1 and it will be recreated on the next setCheckAllLedgersCTime.
3. Check for version mismatch between the component that wrote the znode and the one reading it; upgrade/downgrade consistently.
4. Restore the node from a healthy ZooKeeper snapshot taken before the corruption.

Example fix

// before
long ctime = urManager.getCheckAllLedgersCTime(); // throws on corrupt data
// after
long ctime;
try {
    ctime = urManager.getCheckAllLedgersCTime();
} catch (ReplicationException.UnavailableException e) {
    log.warn("Corrupt checkAllLedgers ctime data, resetting", e);
    urManager.setCheckAllLedgersCTime(System.currentTimeMillis()); // recreate znode
    ctime = urManager.getCheckAllLedgersCTime();
}

// Optional: verify node content sanity via store API before parse-sensitive read
byte[] data = metadataStore.get(checkAllLedgersPath).get().orElse(null);
boolean looksValid = data != null && data.length > 0;
if (!looksValid) {
    log.warn("checkAllLedgers ctime node missing/empty; will be recreated");
}

long ctime;
try {
    ctime = urManager.getCheckAllLedgersCTime();
} catch (ReplicationException.UnavailableException e) {
    if (e.getCause() instanceof RuntimeException) {
        // corrupt data: fall back to sentinel and reset
        ctime = -1;
        urManager.setCheckAllLedgersCTime(System.currentTimeMillis());
    } else {
        throw e; // connectivity/timeout: different remediation
    }
}

When it happens

Trigger: Calling getCheckAllLedgersCTime when the znode contains non-protobuf bytes — data written by an incompatible or older version, manually edited/created znodes, truncated writes, or data created by a different tooling path.

Common situations: Pulsar/BookKeeper version migration where the stored format changed; operators manually creating or restoring znodes from a backup; corruption after an aborted write or bad ZooKeeper snapshot restore.

Related errors

• Failed to initialize BookKeeper metadata
• IOException
• Error contacting with metadata store
• Error parsing proto message
• Error contacting metadata store

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/f8dc9b2414cd6158. Report an issue: GitHub.