apache/pulsar · error · ReplicationException.UnavailableException

Error parsing proto message

Error message

Error parsing proto message

What it means

Thrown when the raw bytes fetched from the metadata store cannot be parsed into the UnderreplicatedLedgerFormat protobuf text format (parseFromTextFormat throws a RuntimeException). This indicates the under-replicated ledger znode/record at the ledger's metadata path is corrupt or was written by an incompatible version, not a connectivity problem.

Source

Thrown at pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java:321

            byte[] data = optRes.get().getValue();

            UnderreplicatedLedgerFormat underreplicatedLedgerFormat = new UnderreplicatedLedgerFormat();

            underreplicatedLedgerFormat.parseFromTextFormat(data);
            PulsarUnderreplicatedLedger underreplicatedLedger = new PulsarUnderreplicatedLedger(ledgerId);
            List<String> replicaList = underreplicatedLedgerFormat.getReplicasList();
            long ctime = (underreplicatedLedgerFormat.hasCtime() ? underreplicatedLedgerFormat.getCtime()
                    : UnderreplicatedLedger.UNASSIGNED_CTIME);
            underreplicatedLedger.setCtime(ctime);
            underreplicatedLedger.setReplicaList(replicaList);
            return underreplicatedLedger;
        } catch (ExecutionException | TimeoutException ee) {
            throw new ReplicationException.UnavailableException("Error contacting with metadata store", ee);
        } catch (InterruptedException ie) {
            Thread.currentThread().interrupt();
            throw new ReplicationException.UnavailableException("Interrupted while connecting metadata store", ie);
        } catch (RuntimeException pe) {
            throw new ReplicationException.UnavailableException("Error parsing proto message", pe);
        }
    }

    @Override
    public CompletableFuture<Void> markLedgerUnderreplicatedAsync(long ledgerId, Collection<String> missingReplicas) {
        log.debug().attr("ledgerId", ledgerId).attr("missingReplicas", missingReplicas)
                .log("markLedgerUnderreplicated");
        final String path = getUrLedgerPath(ledgerId);
        final CompletableFuture<Void> createFuture = new CompletableFuture<>();
        tryMarkLedgerUnderreplicatedAsync(path, missingReplicas, createFuture);
        return createFuture;
    }

    private void tryMarkLedgerUnderreplicatedAsync(final String path,
                                                   final Collection<String> missingReplicas,
                                                   final CompletableFuture<Void> finalFuture) {
        final UnderreplicatedLedgerFormat builder = new UnderreplicatedLedgerFormat();
        if (conf.getStoreSystemTimeAsLedgerUnderreplicatedMarkTime()) {

View on GitHub (pinned to 820761864e)

Solutions

1. Inspect the raw data at the ledger path (e.g. 'get /ledgers/underreplicated/...') and identify the malformed content.
2. Delete the corrupt under-replicated ledger node so the Auditor can re-mark it if the ledger is still under-replicated.
3. Ensure all brokers and the auditor run compatible Pulsar/BookKeeper versions to avoid proto format mismatch.
4. If corruption is widespread, restore the underreplicated-ledger subtree from a consistent backup or trigger a full ledger re-check.

Example fix

// before: repeatedly failing on corrupt node
UnderreplicatedLedger l = urManager.getLedgerUnreplicationInfo(ledgerId); // throws every time
// after: clean the corrupt marker and let re-marking recreate it
try {
    return urManager.getLedgerUnreplicationInfo(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    log.warn("corrupt UR data for {}", ledgerId, e);
    metadataStore.delete(urLedgerPath(ledgerId), Optional.empty()).join();
    return null;
}

// verify the stored payload parses before relying on it
byte[] data = metadataStore.get(urLedgerPath).join().get().getValue();
try {
    new UnderreplicatedLedgerFormat().parseFromTextFormat(data);
} catch (RuntimeException e) {
    // corrupt: quarantine or delete the node before further reads
}

static boolean isParseFailure(ReplicationException.UnavailableException e) {
    Throwable c = e.getCause();
    return c instanceof RuntimeException && !(c instanceof MetadataStoreException);
}

try {
    return urManager.getLedgerUnreplicationInfo(ledgerId);
} catch (ReplicationException.UnavailableException e) {
    if (e.getCause() instanceof RuntimeException) {
        log.warn("corrupt UR marker for {}, deleting node", ledgerId, e);
        metadataStore.delete(urLedgerPath, Optional.empty()).join();
        return null;
    }
    throw e;
}

When it happens

Trigger: Calling getLedgerUnreplicationInfo for a ledger whose stored under-replicated marker contains non-text, truncated, or schema-incompatible data — e.g. the node was written by an older/newer Pulsar with a different proto field layout, or manually edited/corrupted znode data.

Common situations: After a partial metadata-store migration or restore from backup with inconsistent data; manual znode edits via zkCli; version skew between brokers and BookKeeper audit components during a rolling upgrade; disk-level corruption in ZooKeeper transaction logs.

Related errors

• Error while parsing ZK protobuf binary data
• Cursor %s mark-delete position %s is ahead of the last posit
• Malformed configuration file
• Unknown SchemaCompatibilityStrategy.
• Protobuf root message change is not allowed under the '%s' s

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/26b3a75ab17b39b5. Report an issue: GitHub.