apache/pulsar · error · IOException

Invalid MessageIdV5 data: too short

Error message

Invalid MessageIdV5 data: too short

What it means

MessageIdV5.fromByteArray requires at least 12 bytes: 8 bytes for the segmentId (long) plus 4 bytes for the v4-length (int). Shorter arrays cannot contain the fixed header, so it throws IOException('Invalid MessageIdV5 data: too short'). It protects against null/truncated/corrupt message id blobs.

Source

Thrown at pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/MessageIdV5.java:267

                    buf.putInt(seg.getValue().length);
                    buf.put(seg.getValue());
                }
            }
        }
        return buf.array();
    }

    private static Map<Long, byte[]> serializeSegmentVector(Map<Long, MessageId> vector) {
        Map<Long, byte[]> out = new HashMap<>(vector.size());
        for (var entry : vector.entrySet()) {
            out.put(entry.getKey(), entry.getValue().toByteArray());
        }
        return out;
    }

    static MessageIdV5 fromByteArray(byte[] data) throws IOException {
        if (data == null || data.length < 12) {
            throw new IOException("Invalid MessageIdV5 data: too short");
        }
        ByteBuffer buf = ByteBuffer.wrap(data);
        long segmentId = buf.getLong();
        int v4Length = buf.getInt();
        if (v4Length < 0 || v4Length > buf.remaining()) {
            throw new IOException("Invalid MessageIdV5 data: bad v4 length");
        }
        byte[] v4Bytes = new byte[v4Length];
        buf.get(v4Bytes);
        MessageId v4Id = MessageId.fromByteArray(v4Bytes);

        // Section 3: position vector (single-topic / per-segment).
        Map<Long, MessageId> positions = Map.of();
        if (buf.hasRemaining()) {
            positions = readSegmentVector(buf);
        }

        // Section 4: parent topic. Length -1 sentinel means "absent".

View on GitHub (pinned to 820761864e)

Solutions

1. Check data != null && data.length >= 12 before calling fromByteArray.
2. Confirm the bytes were produced by MessageIdV5.toByteArray (v4 ids use a different, shorter format — use the matching deserializer).
3. Re-fetch the message id from the source instead of deserializing the damaged copy.

Example fix

// before
MessageIdV5 id = MessageIdV5.fromByteArray(stored);
// after
if (stored == null || stored.length < 12) {
    throw new IOException("Stored id missing/truncated, re-resolve from source");
}
MessageIdV5 id = MessageIdV5.fromByteArray(stored);

if (data == null || data.length < 12) {
    throw new IOException("Stored message id missing or truncated ("
            + (data == null ? 0 : data.length) + " bytes)");
}

static boolean isPlausibleMessageIdV5(byte[] data) {
    return data != null && data.length >= 12;
}

try {
    MessageIdV5 id = MessageIdV5.fromByteArray(data);
} catch (IOException e) {
    log.warn("Corrupt message id blob, re-resolving from source", e);
}

When it happens

Trigger: Calling MessageIdV5.fromByteArray(null), an empty array, or any array shorter than 12 bytes — typically a truncated id read from a topic, bookkeeper ledger, or custom store.

Common situations: Storing message ids in a fixed-size column too small for the v5 format; cutting/pasting id bytes between formats (e.g. treating a 10-byte legacy id as v5); corruption during serialization to disk.

Related errors

• Invalid MessageIdV5 data: bad parent-topic length
• Invalid checkpoint data: empty
• Invalid MessageIdV5 data: bad v4 length
• Invalid MessageIdV5 data: bad inner id length
• Cannot handle message with null messageId

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/d769e1d5a5ae080b. Report an issue: GitHub.