{"record":{"id":"dd994f65934a0267","repo":"apache/pulsar","slug":"read-ledgermetadata-from-bytes-failed","errorCode":null,"errorMessage":"Read ledgerMetadata from bytes failed","messagePattern":"Read ledgerMetadata from bytes failed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffloadIndexBlockV2Impl.java","lineNumber":239,"sourceCode":"        ledgerInfo.parseFrom(bytes);\n        return ledgerInfo;\n    }\n\n    private OffloadIndexBlockV2 fromStream(DataInputStream dis) throws IOException {\n\n        dis.readInt(); // no used index block length\n        this.dataObjectLength = dis.readLong();\n        this.dataHeaderLength = dis.readLong();\n        while (dis.available() > 0) {\n            long ledgerId = dis.readLong();\n            int indexEntryCount = dis.readInt();\n            int segmentMetadataLength = dis.readInt();\n\n            byte[] metadataBytes = new byte[segmentMetadataLength];\n\n            if (segmentMetadataLength != dis.read(metadataBytes)) {\n                log.error(\"Read ledgerMetadata from bytes failed\");\n                throw new IOException(\"Read ledgerMetadata from bytes failed\");\n            }\n            final LedgerInfo ledgerInfo = parseLedgerInfo(metadataBytes);\n            this.segmentMetadata.put(ledgerId, ledgerInfo);\n            final TreeMap<Long, OffloadIndexEntryImpl> indexEntries = new TreeMap<>();\n\n            for (int i = 0; i < indexEntryCount; i++) {\n                long entryId = dis.readLong();\n                indexEntries.putIfAbsent(entryId, OffloadIndexEntryImpl.of(entryId, dis.readInt(),\n                        dis.readLong(), dataHeaderLength));\n            }\n            this.indexEntries.put(ledgerId, indexEntries);\n        }\n\n        return this;\n    }\n\n    public static int getIndexMagicWord() {\n        return INDEX_MAGIC_WORD;","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffloadIndexBlockV2Impl.java#L221-L257","documentation":"While deserializing a v2 index block, each ledger's protobuf LedgerInfo segment metadata is length-prefixed. If dis.read(metadataBytes) returns fewer bytes than the declared segmentMetadataLength, the stream is truncated or malformed, an error is logged, and an IOException('Read ledgerMetadata from bytes failed') is thrown. The magic word already passed, so this indicates corruption partway into the index.","triggerScenarios":"OffloadIndexBlockV2Impl.fromStream (invoked from get) encounters a declared segmentMetadataLength larger than the remaining bytes in the stream — truncated index blob, interrupted upload, or a length field corrupted by bit rot.","commonSituations":"Failed/incomplete multipart upload of the index object; object storage returning partial reads on flaky networks; index blob damaged during backup/restore.","solutions":["Delete the corrupted index object and re-offload the ledger to rewrite a valid v2 index","Verify the object's stored size vs the expected index length and re-download if truncated","Buffer the full index bytes locally before parsing to rule out short reads from the network layer","Check storage provider integrity (checksum) and restore from a healthy copy if available"],"exampleFix":"// before: parsing directly from a remote stream\nOffloadIndexBlockV2Impl idx = OffloadIndexBlockV2Impl.get(magic, remoteStream);\n// after: read the whole index object first\nbyte[] all = ByteStreams.toByteArray(remoteStream);\nOffloadIndexBlockV2Impl idx = OffloadIndexBlockV2Impl.get(magic,\n    new DataInputStream(new ByteArrayInputStream(all)));","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"IOException last = null;\nfor (int attempt = 0; attempt < 3; attempt++) {\n    try (InputStream in = blob.getPayload().openStream()) {\n        return OffloadIndexBlockV2Impl.get(magic, new DataInputStream(new BufferedInputStream(in)));\n    } catch (IOException e) {\n        last = e;\n        if (!e.getMessage().contains(\"Read ledgerMetadata\")) throw e;\n    }\n}\nthrow last;","preventionTips":["Retry transient short reads from object storage with fresh connections","Compare downloaded bytes to blob content-length before parsing","Alert on repeated index read failures — indicates real corruption, not transient I/O","Keep backup copies of offload index objects"],"tags":["io","truncated-data","corruption","index-block"],"backgroundTag":"truncated-stream","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}