{"record":{"id":"8281241715b39a73","repo":"apache/pulsar","slug":"data-block-header-magic-word-not-match-read-ma","errorCode":null,"errorMessage":"Data block header magic word not match. read: ${magic} expected: ${MAGIC_WORD}","messagePattern":"Data block header magic word not match\\. read: (.+?) expected: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/DataBlockHeaderImpl.java","lineNumber":57,"sourceCode":"    // Payload use this as the start offset.\n    private static final int HEADER_MAX_SIZE = 128;\n    private static final int HEADER_BYTES_USED = 4 /* magic */\n                                               + 8 /* header len */\n                                               + 8 /* block len */\n                                               + 8 /* first entry id */;\n    private static final byte[] PADDING = new byte[HEADER_MAX_SIZE - HEADER_BYTES_USED];\n\n    public static DataBlockHeaderImpl of(int blockLength, long firstEntryId) {\n        return new DataBlockHeaderImpl(HEADER_MAX_SIZE, blockLength, firstEntryId);\n    }\n\n    // Construct DataBlockHeader from InputStream, which contains `HEADER_MAX_SIZE` bytes readable.\n    public static DataBlockHeader fromStream(InputStream stream) throws IOException {\n        CountingInputStream countingStream = new CountingInputStream(stream);\n        DataInputStream dis = new DataInputStream(countingStream);\n        int magic = dis.readInt();\n        if (magic != MAGIC_WORD) {\n            throw new IOException(\"Data block header magic word not match. read: \" + magic\n                    + \" expected: \" + MAGIC_WORD);\n        }\n\n        long headerLen = dis.readLong();\n        long blockLen = dis.readLong();\n        long firstEntryId = dis.readLong();\n        long toSkip = headerLen - countingStream.getCount();\n        if (dis.skip(toSkip) != toSkip) {\n            throw new EOFException(\"Header was too small\");\n        }\n\n        return new DataBlockHeaderImpl(headerLen, blockLen, firstEntryId);\n    }\n\n    private final long headerLength;\n    private final long blockLength;\n    private final long firstEntryId;\n","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/DataBlockHeaderImpl.java#L39-L75","documentation":"DataBlockHeaderImpl.fromStream reads the first 4 bytes of a tiered-storage data block and validates them against the fixed MAGIC_WORD constant. If the leading integer does not match, the stream is not a valid BookKeeper offload data block (or is corrupted/truncated), so an IOException is thrown before any header fields are parsed.","triggerScenarios":"Calling DataBlockHeaderImpl.fromStream on an InputStream whose first 4 bytes are not the data-block magic word: reading a blob that is not a data block (e.g. an index object), an offset past the start of the block, a corrupted/truncated object in blob storage, or an incompatible format version written by a different BookKeeper/Pulsar release.","commonSituations":"Pointing a reader at the wrong object in object storage (offload index instead of payload); manually downloading and re-uploading offloaded ledgers with corruption; restoring offload data from backups with byte shifts; mixing data from incompatible offload format versions.","solutions":["Verify the InputStream starts at offset 0 of an actual offload data block, not an index or metadata object","Check object integrity in blob storage (checksum/ETag) and re-offload the ledger if the block is corrupted","Confirm all brokers use a BookKeeper tiered-storage version with the same data-block format","Enable debug logging on DataBlockHeaderImpl to log the magic value read and compare with the expected constant"],"exampleFix":"// before: reading from an arbitrary stream\nDataBlockHeader header = DataBlockHeaderImpl.fromStream(stream);\n// after: verify format before reading\ntry (InputStream in = blob.getPayload().openStream()) {\n    if (blob.getMetadata().getUserMetadata().get(\"formatVersion\") == null) {\n        throw new IOException(\"Not an offloaded data block: \" + blob.getMetadata().getName());\n    }\n    DataBlockHeader header = DataBlockHeaderImpl.fromStream(in);\n}","handlingStrategy":"validation","validationCode":"// peek the magic word before parsing\nInputStream marked = stream.markSupported() ? stream : new BufferedInputStream(stream);\nmarked.mark(4);\nint magic = new DataInputStream(marked).readInt();\nmarked.reset();\nif (magic != 0x19701226 /* DataBlockHeaderImpl MAGIC_WORD */) {\n    throw new IOException(\"Not a data block, magic=\" + magic);\n}","typeGuard":"static boolean isDataBlockHeader(byte[] first4Bytes) {\n    return first4Bytes != null && first4Bytes.length >= 4\n        && new DataInputStream(new ByteArrayInputStream(first4Bytes)).readInt()\n             == DataBlockHeaderImpl.MAGIC_WORD;\n}","tryCatchPattern":"try {\n    DataBlockHeader h = DataBlockHeaderImpl.fromStream(stream);\n} catch (IOException e) {\n    if (e.getMessage().contains(\"magic word not match\")) {\n        // wrong object or corruption: fetch/re-offload\n    } else { throw e; }\n}","preventionTips":["Always read the data block from its byte 0, never from a cached mid-stream offset","Verify blob user metadata (format version) exists before reading","Enable object-storage checksums to detect corrupted blocks","Keep broker and offload module versions aligned"],"tags":["io","corruption","tiered-storage","bookkeeper"],"backgroundTag":"magic-word-mismatch","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}