apache/pulsar · error · IOException

Invalid MagicWord. read: 0x%x expected: 0x%x or 0x%x

Error message

Invalid MagicWord. read: 0x%x  expected: 0x%x or 0x%x

What it means

OffloadIndexBlockV2BuilderImpl.fromStream reads the leading magic int and dispatches to either the v1 (OffloadIndexBlockImpl) or v2 (OffloadIndexBlockV2Impl) index parser. If the magic matches neither, the stream is not a recognized offload index format, and an IOException listing the read value and both expected magic words is thrown.

Source

Thrown at tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/OffloadIndexBlockV2BuilderImpl.java:125

        streamingOffset = streamingOffset + lastStreamingBlockSize;
        lastStreamingBlockSize = blockSize;

        final List<OffloadIndexEntryImpl> list = entryMap.getOrDefault(ledgerId, new LinkedList<>());
        list.add(OffloadIndexEntryImpl.of(firstEntryId, partId, streamingOffset, dataHeaderLength));
        entryMap.put(ledgerId, list);
        return this;
    }

    @Override
    public OffloadIndexBlockV2 fromStream(InputStream is) throws IOException {
        final DataInputStream dataInputStream = new DataInputStream(is);
        final int magic = dataInputStream.readInt();
        if (magic == OffloadIndexBlockImpl.getIndexMagicWord()) {
            return OffloadIndexBlockImpl.get(magic, dataInputStream);
        } else if (magic == OffloadIndexBlockV2Impl.getIndexMagicWord()) {
            return OffloadIndexBlockV2Impl.get(magic, dataInputStream);
        } else {
            throw new IOException(String.format("Invalid MagicWord. read: 0x%x  expected: 0x%x or 0x%x",
                    magic, OffloadIndexBlockImpl.getIndexMagicWord(),
                    OffloadIndexBlockV2Impl.getIndexMagicWord()));
        }
    }

    @Override
    public OffloadIndexBlock build() {
        checkState(ledgerMetadata != null);
        checkState(!entries.isEmpty());
        checkState(dataObjectLength > 0);
        checkState(dataHeaderLength > 0);
        return OffloadIndexBlockImpl.get(ledgerMetadata, dataObjectLength, dataHeaderLength, entries);
    }

    @Override
    public OffloadIndexBlockV2 buildV2() {
        checkState(!ledgerMetadataMap.isEmpty());
        checkState(true);

View on GitHub (pinned to 820761864e)

Solutions

1. Confirm the object is an offload index written by a compatible Pulsar version; upgrade the client/broker if the index came from a newer format
2. Re-offload the ledger to regenerate a readable index if the blob is corrupted
3. Check the stream is positioned at offset 0 of the index object
4. Log the read magic value and compare with OffloadIndexBlockImpl/OffloadIndexBlockV2Impl getIndexMagicWord() to identify the actual format

Example fix

// before: reading index from an arbitrary object
index = OffloadIndexBlockBuilderImpl.newBuilder().fromStream(magic, wrongStream);
// after: resolve the correct index object key first
String indexKey = DataBlockUtils.parseLedgerId(...); // use the index blob for the ledger
Blob idxBlob = storage.blob(indexKey);
index = OffloadIndexBlockBuilderImpl.newBuilder().fromStream(magic, idxBlob.getPayload().openStream());

DataInputStream in = new DataInputStream(new BufferedInputStream(stream));
in.mark(4);
int magic = in.readInt();
in.reset();
if (magic != OffloadIndexBlockImpl.getIndexMagicWord()
        && magic != OffloadIndexBlockV2Impl.getIndexMagicWord()) {
    throw new IOException("Unknown index format, magic=0x" + Integer.toHexString(magic));
}

try {
    OffloadIndexBlock idx = OffloadIndexBlockBuilderImpl.newBuilder().fromStream(magic, stream);
} catch (IOException e) {
    if (e.getMessage().startsWith("Invalid MagicWord")) {
        log.error("Index object is not a recognized v1/v2 offload index");
    } else { throw e; }
}

When it happens

Trigger: Calling OffloadIndexBlockV2BuilderImpl.fromStream on a stream whose first int is neither INDEX_MAGIC_WORD of v1 nor v2: reading a data block, a future/unknown index format, a corrupted index blob, or a stream positioned mid-object.

Common situations: Offloaded by a newer Pulsar with an index format this reader doesn't know; the bucket key points to payload data instead of the index; index blob corrupted in object storage.

Related errors

• Invalid MagicWord. read: 0x%x expected: 0x%x
• Invalid MagicWord. read: 0x%x expected: 0x%x
• Read ledgerMetadata from bytes failed
• Cursor %s mark-delete position %s is ahead of the last posit
• Data block header magic word not match. read: ${magic} expec

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/9d22c539b060d5d9. Report an issue: GitHub.