apache/pulsar · error · IOException

Data block header magic word not match. read: ${magic} expec

Error message

Data block header magic word not match. read: ${magic} expected: ${MAGIC_WORD}

What it means

DataBlockHeaderImpl.fromStream reads the first 4 bytes of a tiered-storage data block and validates them against the fixed MAGIC_WORD constant. If the leading integer does not match, the stream is not a valid BookKeeper offload data block (or is corrupted/truncated), so an IOException is thrown before any header fields are parsed.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/DataBlockHeaderImpl.java:57

    // Payload use this as the start offset.
    private static final int HEADER_MAX_SIZE = 128;
    private static final int HEADER_BYTES_USED = 4 /* magic */
                                               + 8 /* header len */
                                               + 8 /* block len */
                                               + 8 /* first entry id */;
    private static final byte[] PADDING = new byte[HEADER_MAX_SIZE - HEADER_BYTES_USED];

    public static DataBlockHeaderImpl of(int blockLength, long firstEntryId) {
        return new DataBlockHeaderImpl(HEADER_MAX_SIZE, blockLength, firstEntryId);
    }

    // Construct DataBlockHeader from InputStream, which contains `HEADER_MAX_SIZE` bytes readable.
    public static DataBlockHeader fromStream(InputStream stream) throws IOException {
        CountingInputStream countingStream = new CountingInputStream(stream);
        DataInputStream dis = new DataInputStream(countingStream);
        int magic = dis.readInt();
        if (magic != MAGIC_WORD) {
            throw new IOException("Data block header magic word not match. read: " + magic
                    + " expected: " + MAGIC_WORD);
        }

        long headerLen = dis.readLong();
        long blockLen = dis.readLong();
        long firstEntryId = dis.readLong();
        long toSkip = headerLen - countingStream.getCount();
        if (dis.skip(toSkip) != toSkip) {
            throw new EOFException("Header was too small");
        }

        return new DataBlockHeaderImpl(headerLen, blockLen, firstEntryId);
    }

    private final long headerLength;
    private final long blockLength;
    private final long firstEntryId;

View on GitHub (pinned to 820761864e)

Solutions

Verify the InputStream starts at offset 0 of an actual offload data block, not an index or metadata object
Check object integrity in blob storage (checksum/ETag) and re-offload the ledger if the block is corrupted
Confirm all brokers use a BookKeeper tiered-storage version with the same data-block format
Enable debug logging on DataBlockHeaderImpl to log the magic value read and compare with the expected constant

Example fix

// before: reading from an arbitrary stream
DataBlockHeader header = DataBlockHeaderImpl.fromStream(stream);
// after: verify format before reading
try (InputStream in = blob.getPayload().openStream()) {
    if (blob.getMetadata().getUserMetadata().get("formatVersion") == null) {
        throw new IOException("Not an offloaded data block: " + blob.getMetadata().getName());
    }
    DataBlockHeader header = DataBlockHeaderImpl.fromStream(in);
}

Defensive patterns

Strategy: validation

Validate before calling

// peek the magic word before parsing
InputStream marked = stream.markSupported() ? stream : new BufferedInputStream(stream);
marked.mark(4);
int magic = new DataInputStream(marked).readInt();
marked.reset();
if (magic != 0x19701226 /* DataBlockHeaderImpl MAGIC_WORD */) {
    throw new IOException("Not a data block, magic=" + magic);
}

Type guard

static boolean isDataBlockHeader(byte[] first4Bytes) {
    return first4Bytes != null && first4Bytes.length >= 4
        && new DataInputStream(new ByteArrayInputStream(first4Bytes)).readInt()
             == DataBlockHeaderImpl.MAGIC_WORD;
}

Try / catch

try {
    DataBlockHeader h = DataBlockHeaderImpl.fromStream(stream);
} catch (IOException e) {
    if (e.getMessage().contains("magic word not match")) {
        // wrong object or corruption: fetch/re-offload
    } else { throw e; }
}

Prevention

Always read the data block from its byte 0, never from a cached mid-stream offset
Verify blob user metadata (format version) exists before reading
Enable object-storage checksums to detect corrupted blocks
Keep broker and offload module versions aligned

When it happens

Trigger: Calling DataBlockHeaderImpl.fromStream on an InputStream whose first 4 bytes are not the data-block magic word: reading a blob that is not a data block (e.g. an index object), an offset past the start of the block, a corrupted/truncated object in blob storage, or an incompatible format version written by a different BookKeeper/Pulsar release.

Common situations: Pointing a reader at the wrong object in object storage (offload index instead of payload); manually downloading and re-uploading offloaded ledgers with corruption; restoring offload data from backups with byte shifts; mixing data from incompatible offload format versions.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/8281241715b39a73. Report an issue: GitHub.