apache/pulsar · error · IOException

Invalid MessageIdV5 data: bad v4 length

Error message

Invalid MessageIdV5 data: bad v4 length

What it means

After reading segmentId and a 4-byte v4 length, fromByteArray validates that the length is non-negative and does not exceed the remaining bytes; a value failing either check means the buffer is corrupt or misaligned, so it throws IOException('Invalid MessageIdV5 data: bad v4 length').

Source

Thrown at pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/MessageIdV5.java:273

    }

    private static Map<Long, byte[]> serializeSegmentVector(Map<Long, MessageId> vector) {
        Map<Long, byte[]> out = new HashMap<>(vector.size());
        for (var entry : vector.entrySet()) {
            out.put(entry.getKey(), entry.getValue().toByteArray());
        }
        return out;
    }

    static MessageIdV5 fromByteArray(byte[] data) throws IOException {
        if (data == null || data.length < 12) {
            throw new IOException("Invalid MessageIdV5 data: too short");
        }
        ByteBuffer buf = ByteBuffer.wrap(data);
        long segmentId = buf.getLong();
        int v4Length = buf.getInt();
        if (v4Length < 0 || v4Length > buf.remaining()) {
            throw new IOException("Invalid MessageIdV5 data: bad v4 length");
        }
        byte[] v4Bytes = new byte[v4Length];
        buf.get(v4Bytes);
        MessageId v4Id = MessageId.fromByteArray(v4Bytes);

        // Section 3: position vector (single-topic / per-segment).
        Map<Long, MessageId> positions = Map.of();
        if (buf.hasRemaining()) {
            positions = readSegmentVector(buf);
        }

        // Section 4: parent topic. Length -1 sentinel means "absent".
        String parentTopic = null;
        if (buf.hasRemaining()) {
            int parentLen = buf.getInt();
            if (parentLen >= 0) {
                if (parentLen > buf.remaining()) {
                    throw new IOException("Invalid MessageIdV5 data: bad parent-topic length");

View on GitHub (pinned to 820761864e)

Solutions

1. Regenerate the bytes with MessageIdV5.toByteArray from the same library version.
2. Verify byte order/framing of your storage layer matches the writer (ByteBuffer default big-endian).
3. Sanity-check the total blob length against the expected structure (8 + 4 + v4 + position vector).
4. Re-resolve the message id from the broker instead of trusting the stored blob.

Example fix

// before
byte[] blob = legacyStore.read(key); // may be old v4 format
MessageIdV5 id = MessageIdV5.fromByteArray(blob);
// after
byte[] blob = legacyStore.read(key);
MessageId id = isV5Format(blob) ? MessageIdV5.fromByteArray(blob)
                                : MessageId.fromByteArray(blob);

if (data.length >= 12) {
    int declaredLen = ByteBuffer.wrap(data, 8, 4).getInt();
    if (declaredLen < 0 || 8 + 4 + declaredLen > data.length) {
        throw new IOException("v4 length field inconsistent with blob size");
    }
}

try {
    MessageIdV5 id = MessageIdV5.fromByteArray(data);
} catch (IOException e) {
    log.warn("Bad v5 encoding ({}), attempting legacy v4 parse", e.getMessage());
    MessageId v4 = MessageId.fromByteArray(data);
}

When it happens

Trigger: Deserializing bytes not produced by MessageIdV5.toByteArray: wrong offset/framing, big-endian vs little-endian mismatch, a legacy v4 MessageId blob fed to the v5 parser, or truncated data where the declared length runs past the buffer.

Common situations: Mixing client versions sharing a message-id store; custom persistence writing/reading with differing byte order; an off-by-N offset when concatenating multiple ids into one blob.

Related errors

• Invalid MessageIdV5 data: bad inner id length
• Invalid MessageIdV5 data: too short
• Invalid MessageIdV5 data: bad parent-topic length
• Cannot handle message with null messageId
• payload (${payloadSize} bytes) cannot be decoded with schema

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/aa5d3adb65ebcb1e. Report an issue: GitHub.