What causes "Invalid MagicWord. read: 0x%x expected: 0x%x" in apache/pulsar?

Use OffloadIndexBlockV2BuilderImpl.fromStream to auto-detect v1 vs v2 magic instead of calling the v2 parser directly

apache/pulsar · error · IOException

Invalid MagicWord. read: 0x%x expected: 0x%x

Error message

Invalid MagicWord. read: 0x%x  expected: 0x%x

What it means

OffloadIndexBlockV2Impl.get deserializes the v2 offload index: the supplied magic int must equal the v2 INDEX_MAGIC_WORD. Otherwise the stream is not a v2 index block (wrong format version or corruption), and an IOException showing read vs expected magic is thrown before parsing entries.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffloadIndexBlockV2Impl.java:96

                    .getOrDefault(ledgerId, new TreeMap<>());
            list.forEach(indexEntry -> {
                inLedger.put(indexEntry.getEntryId(), indexEntry);
            });
            block.indexEntries.put(ledgerId, inLedger);
        });

        block.segmentMetadata = metadata;
        block.dataObjectLength = dataObjectLength;
        block.dataHeaderLength = dataHeaderLength;
        return block;
    }

    public static OffloadIndexBlockV2Impl get(int magic, DataInputStream stream) throws IOException {
        OffloadIndexBlockV2Impl block = RECYCLER.get();
        block.indexEntries = Maps.newTreeMap();
        block.segmentMetadata = Maps.newTreeMap();
        if (magic != INDEX_MAGIC_WORD) {
            throw new IOException(String.format("Invalid MagicWord. read: 0x%x  expected: 0x%x",
                    magic, INDEX_MAGIC_WORD));
        }
        block.fromStream(stream);
        return block;
    }

    public void recycle() {
        dataObjectLength = -1;
        dataHeaderLength = -1;
        segmentMetadata = null;
        indexEntries.clear();
        indexEntries = null;
        if (recyclerHandle != null) {
            recyclerHandle.recycle(this);
        }
    }

    @Override

View on GitHub (pinned to 820761864e)

Solutions

Use OffloadIndexBlockV2BuilderImpl.fromStream to auto-detect v1 vs v2 magic instead of calling the v2 parser directly
Confirm the offload format version of the object and use the matching parser
Re-offload the ledger if the index blob is corrupted
Verify the stream starts at the index object's first byte

Example fix

// before: force v2 parsing
OffloadIndexBlockV2Impl idx = OffloadIndexBlockV2Impl.get(magic, stream);
// after: format-agnostic parse
OffloadIndexBlock idx = OffloadIndexBlockBuilderImpl.newBuilder()
    .fromStream(magic, stream); // dispatches on magic word

Defensive patterns

Strategy: try-catch

Validate before calling

int magic = stream.readInt(); // after marking/resetting
if (magic != OffloadIndexBlockV2Impl.getIndexMagicWord()) {
    throw new IOException("Not a v2 offload index, magic=0x" + Integer.toHexString(magic));
}

Try / catch

try {
    OffloadIndexBlockV2Impl idx = OffloadIndexBlockV2Impl.get(magic, stream);
} catch (IOException e) {
    if (e.getMessage().startsWith("Invalid MagicWord")) {
        // likely a v1 index: fall back to OffloadIndexBlockImpl.get
    } else { throw e; }
}

Prevention

Detect format via magic word (use the V2 builder's fromStream) instead of hardcoding v2
Match parser version to the broker version that offloaded the data
Validate object integrity before parsing
Read from byte 0 of the index object

When it happens

Trigger: Calling OffloadIndexBlockV2Impl.get on a stream whose leading int is not the v2 magic — most commonly a v1 index passed directly to the v2 parser, a data block, or a corrupted/truncated index object.

Common situations: Hard-coding the v2 parser for ledgers offloaded by older brokers (v1 format); objects copied/corrupted in storage; reading from the wrong stream offset.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/78cecddc2e5eb542. Report an issue: GitHub.