apache/pulsar · error · IOException

Invalid MagicWord. read: 0x%x expected: 0x%x

Error message

Invalid MagicWord. read: 0x%x  expected: 0x%x

What it means

OffloadIndexBlockImpl.get is the deserialization entry point for the v1 offload index block. The first int read from the stream must equal INDEX_MAGIC_WORD; anything else means the stream is not a v1 offload index (or is corrupt), so an IOException with the read and expected magic words is thrown. It is also dispatched to by OffloadIndexBlockV2BuilderImpl.fromStream when sniffing the format.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffloadIndexBlockImpl.java:82

        this.recyclerHandle = recyclerHandle;
    }

    public static OffloadIndexBlockImpl get(LedgerMetadata metadata, long dataObjectLength,
                                            long dataHeaderLength,
                                            List<OffloadIndexEntryImpl> entries) {
        OffloadIndexBlockImpl block = RECYCLER.get();
        block.indexEntries = Maps.newTreeMap();
        entries.forEach(entry -> block.indexEntries.putIfAbsent(entry.getEntryId(), entry));
        checkState(entries.size() == block.indexEntries.size());
        block.segmentMetadata = metadata;
        block.dataObjectLength = dataObjectLength;
        block.dataHeaderLength = dataHeaderLength;
        return block;
    }

    public static OffloadIndexBlockImpl get(int magic, DataInputStream stream) throws IOException {
        if (magic != INDEX_MAGIC_WORD) {
            throw new IOException(String.format("Invalid MagicWord. read: 0x%x  expected: 0x%x",
                    magic, INDEX_MAGIC_WORD));
        }
        OffloadIndexBlockImpl block = RECYCLER.get();
        block.indexEntries = Maps.newTreeMap();
        block.fromStream(stream);
        return block;
    }

    public void recycle() {
        dataObjectLength = -1;
        dataHeaderLength = -1;
        segmentMetadata = null;
        indexEntries.clear();
        indexEntries = null;
        if (recyclerHandle != null) {
            recyclerHandle.recycle(this);
        }
    }

View on GitHub (pinned to 820761864e)

Solutions

  1. Confirm the object being read is the offload index (offload index key naming convention) not the data block
  2. Re-offload the ledger so a fresh index is written if the index blob is corrupted
  3. Use OffloadIndexBlockV2BuilderImpl.fromStream, which auto-detects v1 vs v2 index magic instead of forcing v1 parsing
  4. Verify the InputStream starts at offset 0 of the index object

Example fix

// before: assume v1
OffloadIndexBlock idx = OffloadIndexBlockImpl.get(magic, stream);
// after: sniff the magic and dispatch
int magic = stream.readInt();
OffloadIndexBlock idx = OffloadIndexBlockBuilderImpl.newBuilder()
    .fromStream(magic, stream);
Defensive patterns

Strategy: try-catch

Validate before calling

DataInputStream in = new DataInputStream(new BufferedInputStream(stream));
in.mark(4);
int magic = in.readInt();
in.reset();
if (magic != OffloadIndexBlockImpl.getIndexMagicWord()) {
    throw new IOException("Not a v1 offload index, magic=0x" + Integer.toHexString(magic));
}

Try / catch

try {
    OffloadIndexBlock idx = OffloadIndexBlockImpl.get(magic, stream);
} catch (IOException e) {
    if (e.getMessage().startsWith("Invalid MagicWord")) {
        // try v2 / auto-detect via OffloadIndexBlockV2BuilderImpl.fromStream
    } else { throw e; }
}

Prevention

When it happens

Trigger: Calling OffloadIndexBlockImpl.get (directly or via fromStream) on a stream whose leading int is not INDEX_MAGIC_WORD: reading a data block instead of the index object, a corrupted index blob, a v2 index handled by a v1-only code path, or reading from the wrong byte offset.

Common situations: Bucket object naming/mapping confusion so the reader fetches payload instead of index; index blob corrupted or truncated to a few bytes; restoring offload data from an incompatible backup or older format.

Related errors

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/c6291f535c8926bc. Report an issue: GitHub.