What causes "Read ledgerMetadata from bytes failed" in apache/pulsar?

Delete the corrupted index object and re-offload the ledger to rewrite a valid v2 index

apache/pulsar · error · IOException

Read ledgerMetadata from bytes failed

Error message

Read ledgerMetadata from bytes failed

What it means

While deserializing a v2 index block, each ledger's protobuf LedgerInfo segment metadata is length-prefixed. If dis.read(metadataBytes) returns fewer bytes than the declared segmentMetadataLength, the stream is truncated or malformed, an error is logged, and an IOException('Read ledgerMetadata from bytes failed') is thrown. The magic word already passed, so this indicates corruption partway into the index.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffloadIndexBlockV2Impl.java:239

        ledgerInfo.parseFrom(bytes);
        return ledgerInfo;
    }

    private OffloadIndexBlockV2 fromStream(DataInputStream dis) throws IOException {

        dis.readInt(); // no used index block length
        this.dataObjectLength = dis.readLong();
        this.dataHeaderLength = dis.readLong();
        while (dis.available() > 0) {
            long ledgerId = dis.readLong();
            int indexEntryCount = dis.readInt();
            int segmentMetadataLength = dis.readInt();

            byte[] metadataBytes = new byte[segmentMetadataLength];

            if (segmentMetadataLength != dis.read(metadataBytes)) {
                log.error("Read ledgerMetadata from bytes failed");
                throw new IOException("Read ledgerMetadata from bytes failed");
            }
            final LedgerInfo ledgerInfo = parseLedgerInfo(metadataBytes);
            this.segmentMetadata.put(ledgerId, ledgerInfo);
            final TreeMap<Long, OffloadIndexEntryImpl> indexEntries = new TreeMap<>();

            for (int i = 0; i < indexEntryCount; i++) {
                long entryId = dis.readLong();
                indexEntries.putIfAbsent(entryId, OffloadIndexEntryImpl.of(entryId, dis.readInt(),
                        dis.readLong(), dataHeaderLength));
            }
            this.indexEntries.put(ledgerId, indexEntries);
        }

        return this;
    }

    public static int getIndexMagicWord() {
        return INDEX_MAGIC_WORD;

View on GitHub (pinned to 820761864e)

Solutions

Delete the corrupted index object and re-offload the ledger to rewrite a valid v2 index
Verify the object's stored size vs the expected index length and re-download if truncated
Buffer the full index bytes locally before parsing to rule out short reads from the network layer
Check storage provider integrity (checksum) and restore from a healthy copy if available

Example fix

// before: parsing directly from a remote stream
OffloadIndexBlockV2Impl idx = OffloadIndexBlockV2Impl.get(magic, remoteStream);
// after: read the whole index object first
byte[] all = ByteStreams.toByteArray(remoteStream);
OffloadIndexBlockV2Impl idx = OffloadIndexBlockV2Impl.get(magic,
    new DataInputStream(new ByteArrayInputStream(all)));

Defensive patterns

Strategy: retry

Try / catch

IOException last = null;
for (int attempt = 0; attempt < 3; attempt++) {
    try (InputStream in = blob.getPayload().openStream()) {
        return OffloadIndexBlockV2Impl.get(magic, new DataInputStream(new BufferedInputStream(in)));
    } catch (IOException e) {
        last = e;
        if (!e.getMessage().contains("Read ledgerMetadata")) throw e;
    }
}
throw last;

Prevention

Retry transient short reads from object storage with fresh connections
Compare downloaded bytes to blob content-length before parsing
Alert on repeated index read failures — indicates real corruption, not transient I/O
Keep backup copies of offload index objects

When it happens

Trigger: OffloadIndexBlockV2Impl.fromStream (invoked from get) encounters a declared segmentMetadataLength larger than the remaining bytes in the stream — truncated index blob, interrupted upload, or a length field corrupted by bit rot.

Common situations: Failed/incomplete multipart upload of the index object; object storage returning partial reads on flaky networks; index blob damaged during backup/restore.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/dd994f65934a0267. Report an issue: GitHub.